Remote customer service system, remote customer service method, and remote customer service program
The remote customer service system addresses the limitation of partial views by integrating multiple camera feeds to enhance operator awareness, enabling comprehensive store coverage and improved service quality.
Patent Information
- Application Number
- JP2024088426
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2024-05-30
- Publication Date
- 2024-11-21
- Estimated Expiration
- 2044-05-30
AI Technical Summary
Existing remote customer service systems fail to provide a comprehensive view of the store situation, missing customers outside the camera's field of view and limiting the operator's ability to offer effective service to all potential customers.
A remote customer service system that utilizes multiple cameras to capture both a customer-facing image and a broader area view, displaying these images simultaneously on an operator's terminal to enhance situational awareness.
Enables operators to provide remote customer service considering the entire store environment, improving service quality by ensuring no customers are overlooked and allowing for proactive engagement.
Smart Images

Figure 0007587779000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a remote customer service system, a remote customer service method, and a remote customer service program. [Background technology]
[0002] Due in part to the impact of the COVID-19 pandemic, digital transformation (DX) is accelerating in most parts of society. DX is particularly prevalent in the customer service industry, which is heavily affected by labor shortages, and the introduction of remote customer service systems is currently in the spotlight as a way to improve productivity and resolve labor shortages.
[0003] In this regard, Patent Document 1 discloses a remote customer service system including a store terminal arranged in a store, an operator terminal, and a management device. In the remote customer service system disclosed in Patent Document 1, thumbnails of images taken by a plurality of store terminals arranged in a plurality of stores are displayed side by side on a standby screen displayed on the operator terminal. When one of the thumbnails of the images is designated by an operator, the store terminal that takes the image of the designated thumbnail and the operator terminal are communicably connected. After that, an image showing the operator taken by the operator terminal is displayed on the store terminal, and an image showing the customer taken by the store terminal is displayed on the operator terminal. In this way, the operator can provide remote customer service to a customer in the store from a point physically distant from the store through the operator terminal and the store terminal. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] Patent Publication No. 2021-153294 Summary of the Invention [Problem to be solved by the invention]
[0005] The remote customer service system disclosed in Patent Document 1 can provide remote customer service in many stores with a small number of operators, and therefore contributes greatly to improving productivity and resolving labor shortages in the customer service industry. On the other hand, remote customer service using the remote customer service system has disadvantages compared to normal face-to-face customer service. In particular, while the video captured by the store terminal can recognize customers present within the field of view of the camera of the store terminal, it cannot recognize customers present outside the field of view of the camera of the store terminal. For this reason, the operator cannot fully grasp the bird's-eye view of the store situation through the store terminal, and therefore cannot provide remote customer service that takes into account the store situation around the store terminal. Furthermore, the operator does not notice the presence of customers outside the field of view who want to be served remotely, and therefore misses opportunities to provide remote customer service to customers who are outside the field of view. Thus, there is room for further improvement of the remote customer service system from the above perspective.
[0006] In view of the above-mentioned points, the present disclosure aims to provide a remote customer service system, a remote customer service method, and a remote customer service program that can improve the quality of remote customer service. [Means for solving the problem]
[0007] A remote customer service system according to one embodiment of the present disclosure is a remote customer service system that supports remote customer service and provides remote customer service using a remote customer service device configured to capture a first video showing a customer, multiple cameras configured to capture a second video showing a state of a predetermined area, and an information terminal operated by an operator providing remote customer service. The remote customer service system identifies at least one camera among the multiple cameras that is associated with the remote customer service device, and causes the first video captured by the remote customer service device and the second video captured by the identified camera to be displayed on the information terminal.
[0008] According to the above configuration, the operator can provide remote customer service that takes into consideration the situation around the remote customer service device by checking both the first image and the second image displayed on the information terminal. In this respect, the operator can check the situation around the remote customer service device through the second image captured by the camera. On the other hand, in the conventional remote customer service system, only the first image captured by the remote customer service device is displayed on the information terminal, so the operator can only check a part of the area in front of the remote customer service device. Therefore, the operator cannot fully grasp the bird's-eye view of the situation through the information terminal. According to the remote customer service system according to the present disclosure, it is possible to provide remote customer service that takes into consideration the bird's-eye view of the situation around the remote customer service device through both the first image and the second image. Therefore, a remote customer service system that can improve the quality of remote customer service can be provided.
[0009] In addition, when multiple cameras are identified as cameras associated with the remote customer service device, the remote customer service system may display thumbnails of multiple second images captured by the identified multiple cameras on the information terminal together with the first image.
[0010] According to the above configuration, it is possible to provide remote customer service that takes into consideration an overall situation around the remote customer service device through the first video and the second video displayed on the information terminal.
[0011] In addition, when one of the thumbnails of the multiple second images is specified in response to an input operation on the information terminal, the remote customer service system may display on the information terminal an enlarged version of the second image corresponding to the specified thumbnail together with the first image.
[0012] According to the above configuration, the operator can check the detailed bird's-eye view of the surroundings of the remote customer service device through the enlarged second image.
[0013] The remote customer service system may include a plurality of remote customer service devices. The remote customer service system may display a plurality of first images captured by the plurality of remote customer service devices on the information terminal, and when a first remote customer service device is designated among the plurality of remote customer service devices in response to an input operation on the information terminal, identify at least one camera associated with the first remote customer service device, and display a first image captured by the first remote customer service device and a second image captured by the identified camera on the information terminal.
[0014] According to the above configuration, when the first remote customer service device is selected from the multiple remote customer service devices by an input operation, the operator can provide remote customer service through the first remote customer service device while taking into consideration the overall situation around the first remote customer service device. This makes it possible to improve the quality of remote customer service provided through the information terminal and the first remote customer service device.
[0015] The remote customer service system may be connected to a first server capable of communicating with the remote customer service device and the information terminal, and a second server capable of communicating with the multiple cameras and the first server. The first video may be provided from the remote customer service device to the information terminal via the first server. The second video may be provided from the camera to the information terminal via the second server and the first server.
[0016] The remote customer service system may also identify the at least one camera associated with the remote customer service device based on identification information of the camera.
[0017] In addition, the remote customer service system may register information indicating a correspondence between the camera and the remote customer service device in response to a user's instruction, and identify at least one of the cameras associated with the remote customer service device based on the information indicating the correspondence.
[0018] The remote customer service system may also identify the at least one camera associated with the remote customer service device by analyzing the second video.
[0019] The remote customer service system may also highlight the remote customer service device shown in the second image.
[0020] According to the above configuration, the operator can clearly recognize the remote customer service device present in the second image.
[0021] The remote customer service system may specify an area in which the remote customer service device exists, and highlight the area in which the remote customer service device exists, in response to an instruction from a user.
[0022] The remote customer service system may also identify an area in which the remote customer service device is present by analyzing the second video, and highlight the area in which the remote customer service device is present.
[0023] The remote customer service system may also highlight people present around the remote customer service device shown in the second image.
[0024] According to the above configuration, the operator can clearly recognize people present around the remote customer service device through the second video.
[0025] The remote customer service system may also specify a surrounding area of the remote customer service device in response to an instruction from a user, and highlight people present in the surrounding area.
[0026] The remote customer service system may also analyze the second video to detect people present in the surrounding area of the remote customer service device, and highlight the people present in the surrounding area.
[0027] The remote customer service system may also execute a predetermined display process on the information terminal in response to an occurrence of a predetermined event in the second video.
[0028] According to the above configuration, the operator can clearly recognize that a predetermined event (for example, the attention area is congested, etc.) has occurred through a predetermined display process in the information terminal.
[0029] The specified event may be a determination condition related to at least one of the number of people present in the attention area of the second video and the duration of their stay.
[0030] According to the above configuration, the occurrence of a specified event in the second video (e.g., the area of interest being crowded) is determined based on judgment conditions related to the number of people present in the area of interest and their staying time, enabling the operator to provide remote customer service that takes into account the number of people present in the area of interest and their staying time.
[0031] The predetermined display process may be to highlight the second video in which the predetermined event has occurred.
[0032] According to the above configuration, the operator can clearly recognize through the highlighted second image that a predetermined event (e.g., the attention area is congested, etc.) has occurred in the second image.
[0033] The predetermined display process may be highlighting a first image corresponding to a second image in which the predetermined event occurs.
[0034] According to the above configuration, the operator can clearly recognize, through the highlighted first image, that a predetermined event is occurring in a store or the like in which the remote customer service device that captures the first image is installed. Therefore, the operator can grasp the necessity of remote customer service in the store, and can appropriately prevent loss of business or sales opportunities due to remote customer service. In this way, a remote customer service system that can improve the quality of remote customer service can be provided.
[0035] The predetermined display process may be to display a second video in which the predetermined event occurs together with a first video corresponding to the second video in which the predetermined event occurs.
[0036] According to the above configuration, the operator can clearly recognize the situation of a store or the like where a specific event is occurring (for example, the area of interest is crowded) through the first and second images, and can immediately grasp the need for remote customer service in that store.
[0037] The information terminal may be configured to capture a third image showing the operator. The remote customer service system may cause the remote customer service device to display the third image captured by the information terminal.
[0038] A remote customer service system according to one aspect of the present disclosure is a remote customer service system that supports remote customer service and provides remote customer service using a plurality of remote customer service devices configured to capture a first video showing a customer, a camera configured to capture a second video showing a state of a predetermined area, and an information terminal operated by an operator providing remote customer service. The remote customer service system identifies at least one remote customer service device associated with the camera among the plurality of remote customer service devices, and causes the second video captured by the camera and the first video captured by the identified remote customer service device to be displayed on the information terminal.
[0039] A remote customer service method according to an embodiment of the present disclosure is executed by a remote customer service system. The remote customer service system supports remote customer service and performs remote customer service using a remote customer service device configured to capture a first video showing a customer, a plurality of cameras configured to capture a second video showing a state of a predetermined area, and an information terminal operated by an operator performing remote customer service. The remote customer service method includes a step of identifying at least one camera among the plurality of cameras associated with the remote customer service device, and a step of displaying the first video captured by the remote customer service device and the second video captured by the identified camera on the information terminal.
[0040] According to the above method, it is possible to provide remote customer service that takes into consideration the overall situation around the remote customer service device through both the first video and the second video, and therefore it is possible to provide a remote customer service method that can improve the quality of remote customer service.
[0041] Also, a remote customer service program may be provided that causes a remote customer service system to execute the remote customer service method. Effect of the Invention
[0042] According to the present disclosure, it is possible to provide a remote customer service system, a remote customer service method, and a remote customer service program that can improve the quality of remote customer service. [Brief description of the drawings]
[0043] [Figure 1] 1 is a diagram showing a remote customer service system according to an embodiment of the present disclosure (hereinafter referred to as the present embodiment). [Diagram 2] FIG. 2 illustrates an example of a hardware configuration of a remote customer service device. [Diagram 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of a camera. [Figure 4] FIG. 2 illustrates an example of a hardware configuration of a first server. [Diagram 5] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information terminal. [Figure 6] 1 is a diagram for explaining the connection relationship between a plurality of remote customer service devices, a plurality of cameras, an information terminal, a first server, and a second server. FIG. [Figure 7] 1 is a flowchart (part 1) for explaining a series of processes executed by the remote customer service system according to the present embodiment. [Figure 8] 11 is a flowchart (part 2) for explaining a series of processes executed by the remote customer service system according to the present embodiment. [Figure 9] 1A is a diagram showing an example of an operator management table, FIG. 1B is a diagram showing an example of a company management table, and FIG. 1C is a diagram showing an example of a store management table. [Figure 10] FIG. 13 is a diagram showing an example of a standby screen on which thumbnails of a plurality of videos captured by a plurality of remote customer service devices are displayed. [Figure 11] FIG. 13 is a diagram showing an example of a customer service screen on which an image of a remote customer service device specified by an operator is displayed. [Figure 12] 13 is a diagram showing an example of a customer service screen on which an image of a remote customer service device specified by an operator and thumbnails of images from a plurality of cameras are displayed. FIG. [Figure 13] FIG. 13 is a diagram showing an example of a customer service screen on which an image from a camera specified by an operator is displayed in an enlarged manner. [Figure 14] 11 is a flowchart for explaining a process of highlighting a thumbnail of an image of a remote customer service device in accordance with congestion in a set attention area in an image from a camera. [Figure 15] A figure showing how one of the thumbnails of images of multiple remote customer service devices is highlighted in response to the congestion of a set area of interest in the camera image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0044] (Configuration of remote customer service system 1) Hereinafter, the remote customer service system 1 according to the present embodiment will be described with reference to the drawings. FIG. 1 is a diagram showing the remote customer service system 1 according to the present embodiment. As shown in FIG. 1, the remote customer service system 1 includes remote customer service devices 2a to 2c, cameras 3a-1 to 3c-10, a first server 4, a second server 5, and information terminals 6e to 6g. These elements are connected to a communication network 7. The communication network 7 is configured by at least one of a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, and a wireless core network. Note that the elements provided in the remote customer service system 1 are not limited to these elements, and other elements not shown in FIG. 1 may be further provided in the remote customer service system 1.
[0045] In the remote customer service system 1, operators Ue to Ug, who are physically distant from the stores, can remotely serve customers who visit each store through the information terminals 6e to 6g. For example, operator Ue can remotely serve customers who visit store A through the information terminal 6e and the remote customer service device 2a installed in store A. Similarly, operator Uf can remotely serve customers who visit store B through the information terminal 6f and the remote customer service device 2b installed in store B. Operator Ug can remotely serve customers who visit store C through the information terminal 6g and the remote customer service device 2c installed in store C. The store may be either indoors or outdoors. In addition, the remote customer service system 1 may be installed not only in stores, but also in various places such as building reception desks, information centers, various transportation counters, event venues, ticket booths, parking lots, warehouses, factories, guardhouses, university campuses, etc. In addition, customers may be people who receive various services in these various places, regardless of whether they are paid or free, depending on the respective places.
[0046] In the following description, for convenience of explanation, the remote customer service devices 2a-2c may be collectively referred to simply as the remote customer service device 2. Similarly, the cameras 3a-1-3c-10 may be collectively referred to simply as the cameras 3, and the information terminals 6e-6g may be collectively referred to simply as the information terminal 6. Furthermore, the operators Ue-Ug who operate the information terminals 6e-6g may be collectively referred to simply as the operator U. Although three remote customer service devices 2, thirty cameras 3, and three information terminals 6 are illustrated in FIG. 1, the numbers of these devices are not particularly limited.
[0047] (Configuration of remote customer service device 2) Next, the hardware configuration of the remote customer service device 2 will be described below with reference to FIG. 2. In this embodiment, the remote customer service devices 2a to 2c have the same configuration. FIG. 2 is a diagram showing an example of the hardware configuration of the remote customer service device 2. The remote customer service devices 2a to 2c are communicably connected to the first server 4 via a communication network 7. The remote customer service device 2 is configured to support remote customer service, and is arranged in each store (e.g., restaurant, retail store, etc.) of the company to which the operators Ue to Ug who perform remote customer service belong. More specifically, the remote customer service device 2a is arranged in store A and configured to support remote customer service for customers who visit store A. The remote customer service device 2b is arranged in store B and configured to support remote customer service for customers who visit store B. The remote customer service device 2c is arranged in store C and configured to support remote customer service for customers who visit store C. Customers who visit the store can receive remote customer service from the operator U through the display unit 26 and speaker 29 of the remote customer service device 2.
[0048] 2, the remote customer service device 2 includes a control unit 21, a storage device 22, an input / output interface 23, a communication unit 24, an input operation unit 25, a display unit 26, an imaging unit 27, a microphone 28, and a speaker 29. These elements are connected to a communication bus 20.
[0049] The control unit 21 includes a memory and a processor. The memory is configured to store computer-readable instructions (programs). For example, the memory is configured from a ROM (Read Only Memory) in which various programs and the like are stored, and a RAM (Random Access Memory) having multiple work areas in which various programs and the like executed by the processor are stored. The processor is configured from at least one of a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and a GPU (Graphics Processing Unit). The CPU may be configured from multiple CPU cores. The GPU may be configured from multiple GPU cores. The processor may be configured to load a program designated from various programs incorporated in the storage device 22 or the ROM onto the RAM and execute various processes in cooperation with the RAM.
[0050] The storage device 22 is, for example, a storage device (storage) such as a hard disk drive (HDD), a solid state drive (SSD), or a flash memory, and is configured to store programs and various data. The input / output interface 23 is an interface that enables a connection between an external device and the remote customer service device 2, and includes an interface conforming to a predetermined communication standard such as the USB standard or the HDMI (registered trademark) standard. The communication unit 24 is configured to connect the remote customer service device 2 to the communication network 7. The communication unit 24 includes, for example, a wireless communication module and / or a wired communication module for wirelessly communicating with an external device such as a wireless base station or a wireless LAN router. The wireless communication module includes a transmitting / receiving antenna and a signal processing circuit. The wireless communication module may be a wireless communication module conforming to a short-range wireless communication standard such as Wi-Fi (registered trademark) or Bluetooth (registered trademark), or may be a wireless communication module conforming to an X-generation mobile communication system (for example, a fourth-generation mobile communication system such as LTE) using a subscriber identity module (SIM).
[0051] The input operation unit 25 is, for example, a touch panel, a mouse, and / or a keyboard arranged over the image display of the display unit 26, and is configured to receive input operations of customers and generate operation signals corresponding to the input operations. The display unit 26 is, for example, configured by an image display and an image display circuit that drives and controls the image display.
[0052] The imaging unit 27 is configured to capture an image of the surrounding environment (particularly, the front area) of the remote customer service device 2. In particular, the imaging unit 27 is configured to generate a video signal indicating the surrounding environment of the remote customer service device 2, and includes an optical system, an image sensor, and an analog processing circuit. The optical system includes, for example, an optical lens and a color filter. The image sensor includes a CCD (Charge-Coupled Device) or a CMOS (Complementary MOS), etc. The analog processing circuit is configured to process the video signal (analog signal) photoelectrically converted by the image sensor, and includes, for example, an amplifier and an AD converter.
[0053] In this embodiment, the imaging unit 27 may be provided on the front surface of the housing 202 of the remote customer service device 2 (see FIG. 6). In this case, the imaging unit 27 can capture an image of a customer present in the front area of the remote customer service device 2. In other words, the field of view of the imaging unit 27 is a part of the front area of the remote customer service device 2, so the remote customer service device 2 cannot capture an image of the surrounding area excluding the front area. The mounting position of the imaging unit 27 on the housing 202 is not particularly limited.
[0054] The microphone 28 is configured to collect voices uttered by customers and generate a voice signal (digital signal) based on the collected voices. The speaker 29 is configured to receive voice data from the control unit 21 and output a voice based on the received voice data to the outside.
[0055] (Camera 3 Configuration) Next, the hardware configuration of camera 3 will be described below with reference to FIG. 3. In this embodiment, cameras 3a-1 to 3c-10 have the same configuration. FIG. 3 is a diagram showing an example of the hardware configuration of camera 3. Camera 3 is configured to acquire video showing its surrounding environment through shooting, and may be placed inside or around a store (an example of a predetermined area). Camera 3 is a network-type surveillance camera that monitors the situation in the store, and is communicatively connected to second server 5 via communication network 7.
[0056] As shown in FIG. 1, each of the cameras 3a-1 to 3a-10 is fixedly installed at a predetermined location in store A and configured to capture an image showing an overview of store A through shooting. Similarly, each of the cameras 3b-1 to 3b-10 is fixedly installed at a predetermined location in store B and configured to capture an image showing an overview of store B through shooting. Each of the cameras 3c-1 to 3c-10 is fixedly installed at a predetermined location in store C and configured to capture an image showing an overview of store C through shooting. Each camera 3 transmits video data to the second server 5 via the communication network 7. The second server 5 stores the video data transmitted by each camera 3.
[0057] 3, the camera 3 includes a control unit 31, a storage device 32, a position information acquisition unit 33, a communication unit 34, an input operation unit 35, an imaging unit 36, and a PTZ mechanism 37, and these elements are connected to a communication bus 30. The camera 3 may also include a built-in battery (not shown). The camera 3 may also include a microphone and a speaker.
[0058] The control unit 31 includes a memory and a processor. The memory is configured to store computer-readable instructions (programs). For example, the memory is composed of a ROM and a RAM. The processor is composed of at least one of a CPU, an MPU, and a GPU. The storage device 32 is, for example, a storage device (storage) such as an HDD, an SSD, or a flash memory, and is configured to store programs and various data. The position information acquisition unit 33 is configured to acquire position information (longitude, latitude) of the camera 3, and is, for example, a GPS (Global Positioning System) receiver.
[0059] The communication unit 34 is configured to connect the camera 3 to the communication network 7. The communication unit 34 includes a wireless communication module and / or a wired communication module for wirelessly communicating with external devices such as a wireless base station or a wireless LAN router. The input operation unit 35 is configured to receive an input operation by an operator and generate an operation signal according to the input operation by the operator. The imaging unit 36 is configured to capture an image of the surrounding environment of the camera 3. In particular, the imaging unit 36 is configured to generate a video signal indicating the surrounding environment of the camera 3, and includes an optical system, an image sensor, and an analog processing circuit.
[0060] The PTZ mechanism 37 includes a pan mechanism, a tilt mechanism, and a zoom mechanism. The pan mechanism is configured to change the orientation of the camera 3 in the horizontal direction. The tilt mechanism is configured to change the orientation of the camera 3 in the vertical direction. The zoom mechanism is configured to enlarge (zoom in) or reduce (zoom out) an image showing an object to be captured by changing the angle of view of the camera 3. The zoom mechanism may optically change the angle of view of the camera 3 by changing the focal length of an optical lens included in the imaging unit 36, or may digitally change the angle of view of the camera 3. In this embodiment, in response to an input operation of the operator U on the information terminal 6, an instruction signal instructing the camera 3 to pan, tilt, and / or tilt is transmitted from the information terminal 6 to the camera 3 via the second server 5. In this case, the control unit 31 drives the PTZ mechanism 37 in response to the received instruction signal, thereby realizing each function (PTZ function) of the camera 3, i.e., pan, tilt, and zoom. In this manner, the PTZ function of the camera 3 can be realized through remote control of the information terminal 6.
[0061] The camera 3 can transmit images (image stream) showing the surrounding environment of the camera 3 to the second server 5 in real time via the communication network 7.
[0062] (Configuration of the first server 4) Next, the hardware configuration of the first server 4 will be described below with reference to Fig. 4. Fig. 4 is a diagram showing an example of the hardware configuration of the first server 4. In this embodiment, the first server 4 may be configured as a WEB server, an application server, and a video data server. In other words, the first server 4 has a function of providing a WEB page, a function of providing a remote customer service application, and a function of providing video data.
[0063] In this respect, the first server 4 is configured to transmit data (e.g., HTML files, CSS files, image / video files, program files, etc.) for displaying the standby screen 100 (see FIG. 10) and the customer service screen 200 (see FIG. 11) on the WEB browser of the information terminal 6. In this manner, the first server 4 functions as a server for providing SaaS (System as a Service). The first server 4 may be configured of multiple servers. The first server 4 may be constructed on-premise or as a cloud server. The first server 4 is communicatively connected to the remote customer service devices 2a-2c and the information terminals 6e-6g via the communication network 7. The first server 4 is configured to transmit video data captured by the remote customer service devices 2a-2c to each of the information terminals 6e-6g.
[0064] As shown in FIG. 4, the first server 4 includes a control unit 41, a storage device 42, an input / output interface 43, and a communication unit 44. These elements are connected to a communication bus 40. The control unit 41 includes a memory and a processor. The memory is configured to store computer-readable instructions. In particular, the memory may store a program for causing the processor to execute a series of processes executed by the first server 4. The memory is configured with a ROM and a RAM. The processor is configured with at least one of a CPU, an MPU, and a GPU.
[0065] The storage device 42 is, for example, a storage device (storage) such as an HDD, SSD, or flash memory, and is configured to store programs and various data. An operator management table, a company management table, and a store management table, which will be described later, are stored in the storage device 42 (see FIG. 9). The storage device 42 may also store images captured by the remote customer service devices 2a to 2c and / or images captured by the information terminals 6e to 6g. The input / output interface 43 is an interface that enables a connection between an external device and the first server 4. The communication unit 44 may include various wired communication modules for communicating with an external device on the communication network 7.
[0066] (Configuration of the second server 5) The second server 5 may have the same hardware configuration as the first server 4 shown in FIG. 4. That is, like the first server 4, the second server 5 includes a control unit, a storage device, an input / output interface, and a communication unit, and these elements are connected to a communication bus. The second server 5 is communicably connected to each of the cameras 3b-1 to 3b-10 via the communication network 7. The second server 5 is configured to receive the video captured by the camera 3, and then store and manage the received video. The second server 5 may store a camera management table for managing the cameras 3 and a video management table for managing the video of the camera 3. The camera management table includes information about the cameras 3 arranged in each store. The video management table includes information about the video data of each camera 3. The second server 5 is communicably connected to the first server 4 through API (Application Programming Interface) linkage. That is, the second server 5 is configured to transmit a desired video to the first server 4 in response to a video transmission request from the first server 4.
[0067] (Configuration of information terminal 6) Next, the hardware configuration of the information terminal 6 will be described below with reference to FIG. 5. In this embodiment, the information terminals 6e to 6g have the same configuration. FIG. 5 is a diagram showing an example of the hardware configuration of the information terminal 6. The information terminal 6 is operated by an operator U who provides remote customer service. The information terminal 6 is configured to capture an image showing the operator U and to acquire the voice of the operator U. The information terminal 6 is disposed at a position physically separated from the remote customer service device 2. In this embodiment, the information terminal 6e is operated by the operator Ue, the information terminal 6f is operated by the operator Uf, and the information terminal 6g is operated by the operator Ug.
[0068] The information terminal 6 may be, for example, a personal computer, a smartphone, a tablet, or a wearable device (for example, AR / VR / MR glasses, etc.) attached to the operator U. The information terminal 6 has a web browser. A remote customer service application provided by the first server 4 may run on the web browser of the information terminal 6. In addition, the remote customer service application may be installed in the information terminal 6 after being downloaded.
[0069] 5, the information terminal 6 includes a control unit 61, a storage device 62, an input / output interface 63, a communication unit 64, an input operation unit 65, a display unit 66, an imaging unit 67, a microphone 68, and a speaker 69. These elements are connected to a communication bus 60.
[0070] The control unit 61 includes a memory and a processor. The memory is configured to store computer-readable instructions. In particular, the memory may store a program for causing the processor to execute a series of processes executed by the information terminal 6. The memory is configured of a ROM and a RAM. The processor is configured of at least one of a CPU, an MPU, and a GPU. The storage device 62 is, for example, a storage device such as an HDD, an SSD, or a flash memory, and is configured to store programs and various data.
[0071] The input / output interface 63 is an interface that enables a connection between an external device and the information terminal 6. The communication unit 64 is configured to connect the information terminal 6 to the communication network 7. The communication unit 64 includes, for example, a wireless communication module and / or a wired communication module. The input operation unit 65 is, for example, a touch panel, a mouse, and / or a keyboard arranged over the image display of the display unit 66, and is configured to receive an input operation of the operator U and generate an operation signal corresponding to the input operation. The display unit 66 is, for example, configured by an image display and an image display circuit that drives and controls the image display.
[0072] The imaging unit 67 is configured to generate a video signal (particularly, a video signal showing the operator U) showing the surrounding environment of the information terminal 6, and includes an optical system, an image sensor, and an analog processing circuit. The microphone 68 is configured to collect the voice of the operator U and generate a voice signal based on the collected voice. The speaker 69 is configured to receive voice data from the control unit 61 and output the voice to the outside based on the received voice data. In this embodiment, after the voice data of the operator U acquired by the microphone 68 is transmitted from the information terminal 6 to the remote customer service device 2 via the communication network 7, the voice of the operator U is output from the speaker 29 of the remote customer service device 2. Similarly, after the voice data of the customer acquired by the microphone 28 is transmitted from the remote customer service device 2 to the information terminal 6 via the communication network 7, the voice of the customer is output from the speaker 69 of the information terminal 6.
[0073] (Remote Customer Service Method According to the Present Embodiment: Flow of a Series of Processing Executed by the Remote Customer Service System 1) Next, a series of processes executed by the remote customer service system 1 according to this embodiment (a remote customer service method according to this embodiment) will be described below with reference mainly to Fig. 6 to Fig. 8. Fig. 6 is a diagram for explaining the connection relationship between the remote customer service devices 2a to 2c, the cameras 3a-1 to 3a-10, the information terminal 6e, the first server 4, and the second server 5. Figs. 7 and 8 are flowcharts for explaining a series of processes executed by the remote customer service system 1.
[0074] In this example, it is assumed that an operator Ue who operates an information terminal 6e (an example of a first information terminal) designates a remote customer service device 2a (an example of a first remote customer service device) and remotely serves a customer in a store A. In this case, it is assumed that an image of the cameras 3a-3 to 3a-8 associated with the remote customer service device 2a among the cameras 3a-1 to 3a-10 arranged in the store A is displayed on the information terminal 6e. As shown in FIG. 6, the first server 4 is communicatively connected to the information terminal 6e and the remote customer service devices 2a to 2c. The first server 4 is communicatively connected to the second server 5 by API linkage. The second server 5 is communicatively connected to the cameras 3a-1 to 3a-10.
[0075] As shown in FIG. 7, in step S1, the information terminal 6e accepts the login information of the operator Ue for logging in to the remote customer service application through the input operation of the operator Ue, and then transmits the login information to the first server 4. The first server 4 executes login authentication of the operator Ue based on the login information of the operator Ue and the operator management table (see FIG. 9(a)). Furthermore, the first server 4 refers to the operator management table to identify the company ID of the company to which the operator Ue belongs, and then refers to the company management table (see FIG. 9(b)) to identify the store ID of each store corresponding to the company ID. After that, the first server 4 refers to the store management table (see FIG. 9(c)) to identify the remote customer service device ID of the remote customer service device 2 corresponding to each store ID. In this way, the first server 4 identifies the remote customer service devices 2a to 2c installed in the stores A to C managed by the operator Ue (step S2). Note that the company here is a broad concept including not only the company itself but also the brand owned by the company. Moreover, the term "store" here refers not only to the store itself but also to a broad concept including the area that the store occupies. Furthermore, a concept of a room may be established for each company, and multiple stores may be associated and managed with each room. For example, in the case of a company that operates multiple stores (or multiple business types), convenience may be improved if the rooms are divided by the area (or business type) under its jurisdiction. In that case, the correspondence between the company ID and the room ID is stored in advance as a room table (not shown), and the company ID in FIG. 9(b) is replaced with the room ID. After the company ID is identified, a list of the rooms corresponding to the company ID is displayed on the information terminal (not shown), and the room ID is identified from the list according to the operator's selection, and the corresponding store ID and remote customer service device ID are identified.
[0076] Next, the first server 4 transmits a video request to the remote customer service devices 2a to 2c (step S3). In response to the video request transmitted from the first server 4, the remote customer service devices 2a to 2c transmit a video (an example of a first video) showing the surrounding environment of the remote customer service device 2 to the first server 4 (step S4). In step S5, the first server 4 receives the video (video stream) captured by the remote customer service devices 2a to 2c, and transmits data (e.g., HTML file, CSS file, image / video file, program file, etc.) for displaying a standby screen 100 (see FIG. 10) to the information terminal 6e. Thereafter, the standby screen 100 is displayed on the display unit 66 of the information terminal 6e (step S6). In particular, the standby screen 100 may be displayed on a WEB browser of the information terminal 6e.
[0077] 10, the standby screen 100 is provided with a video display area 120 and an operator display area 130. Thumbnails of the videos 110a to 110c captured by the remote customer service devices 2a to 2c are displayed in the video display area 120. Information on operators Ue to Ug who are logged in to the remote customer service application and who correspond to the same company ID is displayed in the operator display area 130.
[0078] Next, in step S7, the first server 4 detects the person K in the video 110a captured by the remote customer service device 2a using an image recognition model or the like, and then transmits a highlighting instruction to the information terminal 6e to highlight the thumbnail of the video 110a. After that, in step S8, the information terminal 6e highlights the thumbnail of the video 110a of the remote customer service device 2a in response to the highlighting instruction transmitted from the first server 4 (see FIG. 10). For example, as a visual aspect of the highlighting of the thumbnail of the video 110a, the thumbnail of the video 110a may be highlighted by making the outer frame of the thumbnail of the video 110a a thicker, more noticeable color, or the like.
[0079] In the process of step S6, only the standby screen 100 is displayed on the display unit 66 of the information terminal 6e. However, when the operators Uf and Ug are also logged in to the remote customer service application, the standby screen 100 is also displayed on the display unit 66 of the information terminals 6e and 6f. That is, the first server 4 transmits data for displaying the standby screen 100 to each of the information terminals 6e to 6g. Similarly, in the process of step S7, a highlighting instruction for highlighting the thumbnail of the video 110a is transmitted to each of the information terminals 6e to 6g. Thereafter, in step S8, each of the information terminals 6e to 6g highlights the thumbnail of the video 110a in response to the highlighting instruction.
[0080] Next, in step S9, when the operator Ue wants to provide remote customer service in the store A, the operator Ue designates the thumbnail of the video 110a displayed on the standby screen 100 through the input operation unit 65. Specifically, as shown in FIG. 10, the thumbnail of the video 110a is double-clicked. At this time, the information terminal 6e accepts the designation of the remote customer service device 2a in response to the input operation of the operator Ue, and transmits the designation information of the remote customer service device 2a to the first server 4 (step S10). Next, the first server 4 transmits data (e.g., HTML file, CSS file, image / video file, program file, etc.) for displaying the customer service screen 200 (see FIG. 11) of the remote customer service device 2a in response to the designation information (step S11). After that, the customer service screen 200 is displayed on the display unit 66 of the information terminal 6e (step S12).
[0081] As shown in FIG. 11, the customer service screen 200 includes a video display area 210, a store manual display area 212, an operator video display area 220, a store explanatory material list area 230, and a chat display area 240. In the video display area 210, a video 110a captured by the remote customer service device 2a is displayed. In the store manual display area 212, a customer service manual for the store A is displayed. In the operator video display area 220, a video 223e showing the operator Ue captured by the imaging unit 67 of the information terminal 6e is displayed. In the store explanatory material list area 230, a list of explanatory materials for the store A is displayed, and the content to be displayed on the display unit of the remote customer service device can be selected from the list and displayed according to the operator's instructions. In the chat display area 240, the contents of the chat between the operator Ue and a customer in the store A are displayed.
[0082] The customer service screen 200 is also provided with a home button 260, a camera button 250, a hold button 265, and a customer service end button 263. When the home button 260 is operated by the operator Ue, the customer service screen 200 shown in FIG. 11 is displayed. When the camera button 250 is operated by the operator Ue, the customer service screen 200 shown in FIG. 12 is displayed. When the hold button 265 is operated by the operator Ue, remote customer service to a customer in the store A is put on hold. In this case, the display of the video showing the operator Ue on the remote customer service device 2a may be interrupted. When the customer service end button 263 is operated by the operator Ue, the screen displayed on the information terminal 6e is switched from the customer service screen 200 to the standby screen 100 shown in FIG. 10.
[0083] Next, in step S13, the information terminal 6e transmits a video 223e (an example of a third video) showing the operator Ue, which is photographed by the information terminal 6e, to the remote customer service device 2a. After that, the video 223e (video stream) photographed by the information terminal 6e is displayed on the display unit 26 of the remote customer service device 2a (step S14). In this way, when the operator Ue performs remote customer service in the store A, the first server 4 starts communication between the information terminal 6e and the remote customer service device 2a, so that the video 110a including the customer photographed by the remote customer service device 2a is displayed on the information terminal 6e, and the video 223e including the operator photographed by the information terminal 6e is displayed on the remote customer service device 2a. In this case, the remote customer service device 2a may transmit the video 110a to the information terminal 6e via the first server 4, or may transmit the video 110a directly to the information terminal 6e by P2P (Peer to Peer) without the first server 4. Similarly, the information terminal 6e may transmit the video 223e to the remote customer service device 2a via the first server 4, or may transmit the video 223e directly to the remote customer service device 2a by P2P without going through the first server 4.
[0084] Similarly to the video data, audio data is also transmitted and received between the information terminal 6e and the remote customer service device 2a. That is, the customer's audio data acquired by the microphone 28 of the remote customer service device 2a is output from the speaker 69 of the information terminal 6e, and the operator Ue's audio data acquired by the microphone 68 of the information terminal 6e is output from the speaker 29 of the remote customer service device 2a.
[0085] Next, in step S15, the information terminal 6e transmits a request to display the camera image in response to an input operation by the operator Ue to the first server 4. Specifically, as shown in FIG. 11, the camera button 250 provided on the customer service screen 200 is operated by the operator Ue, whereby the request to display the camera image is transmitted from the information terminal 6e to the first server 4.
[0086] In step S16, the first server 4, in response to the request to display the camera video, transmits a request to transmit the camera video associated with the remote customer service device 2a to the second server 5. Here, the request to transmit the camera video may include information about the cameras 3a-1 to 3a-10 installed in the store A (e.g., the camera IDs of the cameras 3a-1 to 3a-10). In this case, the first server 4 refers to the store management table (FIG. 9(c)) to obtain information about the cameras 3a-1 to 3a-10 installed in the store A, and then transmits a request to transmit the camera video including the information about the cameras 3a-1 to 3a-10 installed in the store A to the second server 5.
[0087] In step S17, the second server 5 identifies the cameras 3a-3 to 3a-8 associated with the remote customer service device 2a in response to the camera video transmission request transmitted from the first server 4. Here, the camera 3 associated with the remote customer service device 2a may be defined as the camera 3 disposed in the vicinity of the remote customer service device 2a and / or the camera 3 that captures video including the presence of the remote customer service device 2a.
[0088] (First example of a process for identifying the cameras 3a-3 to 3a-8 associated with the remote customer service device 2a) In the first example, the remote customer service device 2a and the cameras 3a-3 to 3a-8 are automatically associated with each other based on the camera names. Specifically, the second server 5 may specify the names of the cameras 3a-1 to 3a-10 based on the camera IDs of the cameras 3a-1 to 3a-10 included in the camera video transmission request and on data indicating the correspondence between the camera IDs and the camera names (an example of the camera identification information). Next, the second server 5 may specify the cameras 3a-3 to 3a-8 associated with the remote customer service device 2a based on the names of the cameras 3a-1 to 3a-10. For example, when the names of the cameras 3a-3 to 3a-8 include information on the remote customer service device 2a (for example, a character string indicating the remote customer service device 2a), the cameras 3a-3 to 3a-8 may be specified as the cameras associated with the remote customer service device 2a. The data indicating the correspondence between the camera IDs and the camera names (an example of the camera identification information) may be registered in advance by the operator U. In this example, the camera 3 associated with the remote customer service device 2 is identified based on the camera name as an example of the identification information of the camera 3, but the identification information of the camera 3 is not limited to the camera name.
[0089] (Second Example of Processing for Identifying Cameras 3a-3 to 3a-8 Associated with Remote Customer Service Device 2a) In the second example, the remote customer service device 2a and the cameras 3a-3 to 3a-8 are manually associated with each other based on a user instruction. The user instruction is assumed to be mainly given from the information terminal 6 of the operator U, but may also be given from an information processing device (not shown) of the company that manages them. Specifically, the request to transmit the camera image may include information on the remote customer service device 2a (e.g., the ID of the remote customer service device 2a). In this case, the second server 5 may identify the cameras 3a-3 to 3a-8 associated with the remote customer service device 2a based on the ID of the remote customer service device 2a included in the request to transmit the camera image and data indicating the correspondence between the multiple remote customer service devices 2 and the multiple cameras 3.
[0090] The data indicating the correspondence between the multiple remote customer service devices 2 and the multiple cameras 3 may indicate the correspondence between the IDs of the multiple remote customer service devices 2 and the multiple camera IDs. The data indicating the correspondence includes information on the multiple cameras 3 associated with each remote customer service device 2. For example, the data indicating the correspondence may indicate the correspondence between the ID of the remote customer service device 2a and the IDs of the cameras 3a-3 to 3a-8. In this way, it is possible to identify the cameras 3a-3 to 3a-8 associated with the remote customer service device 2a based on the data indicating the correspondence. The data indicating the correspondence between the multiple remote customer service devices 2 and the multiple cameras 3 may be registered in advance by the operator U.
[0091] (Third Example of Processing for Identifying Cameras 3a-3 to 3a-8 Associated with Remote Customer Service Device 2a) In the third example, the remote customer service device 2a and the cameras 3a-3 to 3a-8 are automatically associated with each other based on video analysis. Specifically, the camera video transmission request may include information about the cameras 3a-1 to 3a-10 installed in the store A (for example, the camera IDs of the cameras 3a-1 to 3a-10). In this case, the second server 5 acquires the videos captured by the cameras 3a-1 to 3a-10 based on the camera IDs of the cameras 3a-1 to 3a-10 included in the camera video transmission request. Then, the second server 5 analyzes the videos captured by the cameras 3a-1 to 3a-10 to extract the videos showing the remote customer service device 2a from the videos captured by the cameras 3a-1 to 3a-10. Then, the second server 5 specifies the cameras 3a-3 to 3a-8 capturing the videos showing the presence of the remote customer service device 2a as the cameras associated with the remote customer service device 2a. In addition, the second server 5 may use an image recognition model that is a trained model constructed by machine learning when extracting the video showing the remote customer service device 2a. In this case, the image recognition model may be constructed from training data that includes image data and information about the remote customer service device 2a included in the image data.
[0092] Next, in step S18, the second server 5 transmits the images (thumbnail display) of the cameras 3a-3 to 3a-8 associated with the remote customer service device 2a to the first server 4. Next, the first server 4 transmits the images of the cameras 3a-3 to 3a-8 to the information terminal 6e (step S19). After that, as shown in FIG. 12, the information terminal 6e displays thumbnails of the images 320a to 320f (examples of the second images) captured by the cameras 3a-3 to 3a-8 on the customer service screen 200 (step S20). Note that if only one camera is identified in step S17, FIG. 12 may be omitted and FIG. 13 may be displayed. In that case, steps S18 to S22 may be omitted, and the image of the identified camera may be enlarged and displayed in the camera image display area 310.
[0093] The customer service screen 200 shown in FIG. 12 includes a video display area 210, a store manual display area 212, and a camera video display area 310. In the camera video display area 310, thumbnails of videos 320a to 320f captured by the cameras 3a-3 to 3a-8 are displayed side by side. Here, the video 320a is the video captured by the camera 3a-3. The video 320f is the video captured by the camera 3a-8. In this way, on the customer service screen 200 shown in FIG. 12, the video 110a captured by the remote customer service device 2a and thumbnails of the videos 320a to 320f captured by the cameras 3a-3 to 3a-8 associated with the remote customer service device 2a are simultaneously displayed on the same screen.
[0094] As shown in FIG. 12, the remote customer service device 2a and the person K present around the remote customer service device 2a are highlighted in the images 320a to 320f. For example, in the image 320b captured by the camera 3a-4, a rectangular mark W1 surrounding the presence area of the remote customer service device 2a is displayed so that the presence area of the remote customer service device 2a is highlighted. Similarly, a rectangular mark W2 surrounding the person K present near the remote customer service device 2a is displayed in the image 320b so that the person K is highlighted. Note that the rectangular mark W2 may be displayed simply when the person K is detected in the image, and may be highlighted with an attention color or the like when the person K enters the vicinity of the remote customer service device 2a.
[0095] (Display process of rectangular mark W1 surrounding the area where the remote customer service device 2a exists in the camera image) For example, the presence area of the remote customer service device 2a may be set according to an input operation of the operator U on the information terminal 6. Although such user instructions are assumed to be mainly given from the information terminal 6 of the operator U, they may also be given from an information processing device (not shown) of the company that manages them. For example, when the operator U specifies the presence area of the remote customer service device 2a in the video 320b while the video 320b is displayed on the information terminal 6, information indicating the presence area of the remote customer service device 2a in the video 320b (for example, coordinate information defining the presence area in the video 320b) is transmitted from the information terminal 6 to the second server 5. After that, the second server 5 specifies the presence area of the remote customer service device 2a based on the information indicating the presence area, and displays a rectangular mark W1 surrounding the presence area of the remote customer service device 2a in the video 320b. After that, a thumbnail of the video 320b on which the rectangular mark W1 is displayed is displayed on the customer service screen 200. Furthermore, the rectangular mark W1 is set and displayed in each of the videos 320a, 320d, and 320e in the same manner as above. Furthermore, the second server 5 may analyze the image 320b using an image recognition model to identify the presence area of the remote customer service device 2a shown in the image 320b, and then display a rectangular mark W1 surrounding the identified presence area of the remote customer service device 2a.
[0096] (Display process of rectangular mark W2 surrounding person K in camera image) Also, the peripheral area of the remote customer service device 2a may be set according to an input operation of the operator U on the information terminal 6. Although such user instructions are assumed to be mainly given from the information terminal 6 of the operator U, they may also be given from an information processing device (not shown) of a company that manages them. For example, when the operator U specifies the peripheral area of the remote customer service device 2a in the video 320b while the video 320b is displayed on the information terminal 6, information indicating the peripheral area of the remote customer service device 2a in the video 320b (for example, coordinate information defining the peripheral area in the video 320b) is transmitted from the information terminal 6 to the second server 5. The peripheral area may be set automatically or manually to an area within a predetermined distance from the remote customer service device 2a or an area where people have stayed or the like occurred with a predetermined frequency or more in the past. Thereafter, the second server 5 sets the peripheral area in the video 320b based on the information transmitted from the information terminal 6. Thereafter, when the second server 5 detects that the person K exists in the surrounding area, the second server 5 displays a rectangular mark W2 surrounding the person K on the video 320b to highlight the person K. Then, a thumbnail of the video 320b in which the rectangular mark W2 is displayed is displayed on the customer service screen 200. Note that the rectangular mark W2 may be displayed simply when the person K is detected in the video, and the rectangular mark W2 may be highlighted with a focus color or the like when the person K is detected to exist in the surrounding area. Furthermore, in the same manner as above, the surrounding area of the remote customer service device 2a is set in each of the videos 320a, 320d, and 320e, and the rectangular mark W2 surrounding the person K existing in the set surrounding area is displayed. Note that the second server 5 may use an image recognition model when detecting that the person K exists in the surrounding area. In this case, the image recognition model may be constructed from learning data consisting of image data and information about the person contained in the image data. In particular, the second server 5 may analyze the image 320b using an image recognition model to detect a person present within the surrounding area of the remote customer service device 2a, and then display a rectangular mark W2 surrounding the detected person on the image 320b.
[0097] In this way, since the rectangular marks W1 and W2 are displayed in some of the images 320a to 320f, the operator Ue can clearly recognize the remote customer service device 2a present in each image by the rectangular mark W1. Similarly, the operator Ue can clearly recognize the person K present around the remote customer service device 2a shown in each image by the rectangular mark W2. Therefore, the operator Ue can visually recognize the remote customer service device 2a and the person K present around it, and can instantly grasp the surrounding situation of the remote customer service device 2a. In this embodiment, the remote customer service device 2a and the person K in the image of the camera 3 are highlighted by the rectangular marks W1 and W2, but the visual manner of highlighting the remote customer service device 2a and the person K is not limited to the rectangular marks W1 and W2.
[0098] According to the present embodiment, the operator Ue can provide remote customer service that takes into consideration the store situation around the remote customer service device 2a by checking both the image 110a (an example of a first image) and thumbnails of the images 320a to 320f (an example of a second image) displayed on the information terminal 6e. In this respect, the operator Ue can check the store situation around the remote customer service device 2a through the images 320a to 320f. On the other hand, in the past, since only the image captured by the remote customer service device 2a was displayed on the information terminal 6e, the operator Ue could only check a part of the area in front of the remote customer service device 2a. For this reason, the operator Ue could not fully grasp the store situation from a bird's-eye view through the information terminal 6e. According to the remote customer service system 1 according to the present disclosure, it is possible to provide remote customer service that takes into consideration the store situation around the remote customer service device 2a through both the image 110a and the images 320a to 320f. Therefore, a remote customer service system 1 that can improve the quality of remote customer service can be provided.
[0099] 8, in step S21, the information terminal 6e receives designation of a thumbnail of the video 320b of the camera 3a-4 in response to an input operation by the operator Ue, and then transmits a request for enlarged display of the video 320b to the first server 4. For example, as shown in FIG. 12, the thumbnail of the video 320b is double-clicked.
[0100] In step S22, the first server 4 transmits a transmission request for the enlarged image 320b of the camera 3a-4 to the second server 5 in response to the enlarged display request from the information terminal 6e. In step S23, the second server 5 transmits the enlarged image 320b to the first server 4 in response to the transmission request from the first server 4. Thereafter, the first server 4 transmits the enlarged image 320b to the information terminal 6e (step S24). As a result, as shown in the customer service screen 200 of FIG. 13, the enlarged image 320b is displayed on the information terminal 6e (step S25). Note that the enlarged display here means the display of the original image relative to the thumbnail of the image 320b, but may also be an image in which a part of the original image (for example, an area including the remote customer service device 2a and the customer) is zoomed in.
[0101] In the customer service screen 200 shown in FIG. 13, a viewer for the video 320b is displayed in the camera video display area 310. The viewer for the video 320b has the enlarged video 320b, a timeline 350 indicating the playback time of the video 320b, and a slider 340 that can slide on the timeline 350. The operator Ue can change the playback time of the video 320b by moving the slider 340 on the timeline 350 of the viewer for the video 320b. For example, when the operator Ue moves the slider 340 to the position of 10:30, the information terminal 6e transmits a transmission request for the video 320b in the time zone before and after 10:30 to the first server 4. In response to the transmission request, the first server 4 acquires the video 320b in the time zone before and after 10:30 from the second server 5, and transmits the acquired video 320b to the information terminal 6e. In this way, the video 320b from the time period around 10:30 is displayed in the camera video display area 310.
[0102] In this way, the operator Ue can check the store situation around the remote customer service device 2a (especially the state of people near the remote customer service device 2a) in detail through both the image 100a and the enlarged image 320b. Therefore, it is possible to provide remote customer service that takes into account the store situation around the remote customer service device 2a, and a remote customer service system 1 that can improve the quality of remote customer service can be provided.
[0103] (Remote Customer Service Method According to a Modification of the Present Embodiment: Processing for Highlighting Thumbnail of Video 110a Depending on Congestion in the Attention Area) Next, mainly with reference to Fig. 14 and Fig. 15, a process (a remote customer service method according to a modified example of this embodiment) of highlighting the thumbnail of the video 110a of the remote customer service device 2a according to the congestion of the attention area S (see Fig. 15) set in the video 320a captured by the camera 3a-3 will be described below. Fig. 14 is a flowchart for explaining the process of highlighting the thumbnail of the video 110a of the remote customer service device 2a according to the congestion of the attention area S set in the video 320a of the camera 3a-3. Fig. 15 is a diagram showing a state in which the video 110a is highlighted among the thumbnails of the videos 110a-110c of the remote customer service devices 2a-2c according to the congestion of the attention area S set in the video 320a of the camera 3a-3.
[0104] 7, in steps S7 and S8, the thumbnail of the video 110a is highlighted in response to detection of a person K in the video 110a. On the other hand, in this modified example, even if a person K is not detected in the video 110a, the thumbnail of the video 110a is highlighted, which is different from the processing in steps S7 and S8. In this respect, in this modified example, the thumbnail of the video 110a is highlighted in response to the state of the attention area S set in the video 320a of the camera 3a-3 associated with the remote customer service device 2a, which is different from the processing in steps S7 and S8.
[0105] In Fig. 14, steps S31 to S34 are similar to steps S1 to S4 in Fig. 7, and therefore description thereof will be omitted. In step S35, the first server 4 receives the images captured by the remote customer service devices 2a to 2c, and then transmits data for displaying a standby screen 100 (see Fig. 15) to the information terminal 6e. Thereafter, the standby screen 100 shown in Fig. 15 is displayed on the display unit 66 of the information terminal 6e (step S36).
[0106] 15, standby screen 100 is provided with video display area 120 and operator display area 130. Thumbnails of videos 110a to 110c captured by remote customer service devices 2a to 2c are displayed in video display area 120. At this point, the outer frame of the thumbnail of video 110a is not highlighted, and video 320a is not displayed as a wipe video superimposed on video 110a.
[0107] Next, in step S37, the second server 5 determines that the attention area S is crowded (an example of a predetermined event) based on a judgment condition associated with the attention area S (see FIG. 15) set in the video 320a. After that, the second server 5 transmits a highlighting instruction including information about the camera 3a-3 capturing the video 320a (e.g., the ID of the camera 3a-3) to the first server 4. Here, the attention area S may be set in the periphery of the remote customer service device 2a and outside the field of view of the imaging unit 27 of the remote customer service device 2a.
[0108] Furthermore, the attention area S in the video 320a may be set in response to an input operation of the operator U on the information terminal 6. For example, when the operator U specifies the attention area S in the video 320a while the video 320a is displayed on the information terminal 6, information indicating the attention area S in the video 320a (for example, coordinate information defining the attention area S in the video 320a) is transmitted from the information terminal 6 to the second server 5. Thereafter, the second server 5 sets the attention area S in the video 320a. Note that the attention area S may be set automatically or manually to an area within a predetermined distance from the remote customer service device 2a or an area where people have been staying or the like with a predetermined frequency or more in the past.
[0109] Further, the judgment condition associated with the attention area S is a judgment condition related to at least one of the number of people present in the attention area S and the staying time. For example, the judgment condition may be a judgment condition indicating that the number of people present in the attention area S is N or more (N is a natural number) and the staying time of the people in the attention area S is M seconds or more (M is a natural number). In this case, the second server 5 may determine that the attention area S is crowded when detecting that the number of people present in the attention area S set in the video 320a is N or more and the staying time is M seconds or more. Here, the second server 5 may use an image recognition model when detecting a person K present in the attention area S. In this example, the congestion of the attention area S is given as an example of a predetermined event in the attention area S, but the predetermined event is not limited thereto.
[0110] Returning to FIG. 14, in step S38, the first server 4 identifies the remote customer service device 2a associated with the camera 3a-3 based on the highlighting instruction from the second server 5 and the store management table (see FIG. 9(c)). Next, the first server 4 transmits an emphasis display instruction to the information terminal 6e to emphasize the thumbnail of the video 110a captured by the remote customer service device 2a (step S39). After that, in step S40, the information terminal 6e emphasizes the thumbnail of the video 110a of the remote customer service device 2a in response to the emphasis display instruction transmitted from the first server 4 (see FIG. 15). For example, as a visual aspect of the emphasis display of the thumbnail of the video 110a, the thumbnail of the video 110a may be emphasized by making the outer frame of the thumbnail of the video 110a thick and conspicuous. Note that steps S40 and after are similar to steps S15 to S25 in FIG. 8, and therefore description thereof will be omitted.
[0111] As shown in Fig. 15, in a state where the thumbnail of the image 110a is highlighted, the image 320a may be superimposed on the image 110a as a wipe image. In this case, in step S37, the second server 5 may transmit the image 320a to the first server 4 together with a highlighting instruction including information about the camera 3a-3. The wipe image may be displayed not only by superimposing the image 110a, but also by a push notification or a pop-up notification. The wipe image may be displayed under the conditions for highlighting in the example shown in Fig. 10.
[0112] According to this modification, in response to detection of congestion in the attention area S in the video 320a, the remote customer service device 2a associated with the camera 3a-3 that captures the video 320a is identified, and the thumbnail of the video 110a captured by the remote customer service device 2a is highlighted. In this way, the operator Ue can clearly recognize that congestion is occurring in the attention area S of the store A in which the remote customer service device 2a is located through the highlighted thumbnail of the video 110a. Therefore, even if there is no person within the field of view of the remote customer service device 2a, the operator Ue can grasp the necessity of remote customer service in the store A, and can appropriately prevent loss of business or sales opportunities due to remote customer service. In this way, a remote customer service system 1 that can improve the quality of remote customer service can be provided.
[0113] In addition, in this modification, since the image 320a is superimposed on the image 110a, the operator Ue can recognize that congestion is occurring in the attention area S of the store A by visually checking the images 320a and 110a simultaneously displayed on the information terminal 6e. In this way, the operator Ue can immediately grasp the necessity of remote customer service in the store A.
[0114] In this modified example, only the standby screen 100 is displayed on the display unit 66 of the information terminal 6e. However, when the operators Uf and Ug are also logged in to the remote customer service application, the standby screen 100 is also displayed on the display units 66 of the information terminals 6e and 6f. That is, in step S35, the first server 4 transmits data for displaying the standby screen 100 to each of the information terminals 6e to 6g. Similarly, in step S39, a highlighting instruction for highlighting the thumbnail of the video 110a is transmitted to each of the information terminals 6e to 6g. Thereafter, in step S40, each of the information terminals 6e to 6g highlights the thumbnail of the video 110a in response to the highlighting instruction.
[0115] In addition, in this modification, the attention area S is set only in the image 320a, but the attention area S may also be set in other camera images (specifically, images 320b to 320f) other than the image 320a. That is, a plurality of attention areas S may be set in the store A. Furthermore, the image 110a displayed on the standby screen 100 is highlighted, but the thumbnail of the image 320a displayed on the customer service screen 200 shown in FIG. 12 may be highlighted. In this case, in step S39 of FIG. 14, the first server 4 transmits a highlighting instruction to highlight the thumbnail of the image 320a to the information terminal 6e. Thereafter, in step S40, the information terminal 6e highlights the thumbnail of the image 320a on the customer service screen 200 in response to the highlighting instruction transmitted from the first server 4. Similarly, when the thumbnail of the image 320a displayed on the customer service screen 200 is highlighted, the operator Ue can clearly recognize that congestion is occurring in the attention area S.
[0116] (Modification in which the image captured by the remote customer service device 2 and the image captured by the camera 3 are displayed interchangeably) In the above description, the image captured by the remote customer service device 2 is displayed as the main image, and the image captured by the camera 3 is displayed as the sub image. However, the main / sub image relationship may be reversed. That is, the image captured by the camera 3 may be displayed as the main image, while the image captured by the remote customer service device 2 may be displayed as the sub image. This is based on the premise that the operator U views the image captured by the camera 3 on the information terminal 6 via a viewer or the like, and displays the image of the camera 3 and the image of the remote customer service device 2 corresponding to the camera 3. In this case, in the description from FIG. 6 onwards, the remote customer service device 2 and the camera 3, and the first server 4 and the second server 5 are respectively replaced. Also, in the data structure of FIG. 9(c), the remote customer service device ID and the camera ID are replaced. In the UI diagrams of FIG. 10 to FIG. 13 and FIG. 15, the image captured by the remote customer service device 2 and the image captured by the camera 3 are replaced.
[0117] Although the embodiment of the present invention has been described above, the technical scope of the present invention should not be interpreted as being limited by the description of the embodiment. The embodiment is an example, and it is understood by those skilled in the art that various modifications of the embodiment are possible within the scope of the invention described in the claims. The technical scope of the present invention should be determined based on the scope of the invention described in the claims and its equivalents. [Explanation of symbols]
[0118] 1: remote customer service system, 2, 2a, 2b, 2c: remote customer service device, 3, 3a-1 to 3c-10: camera, 4: first server, 5: second server, 6, 6e, 6f, 6g: information terminal, 7: communication network, 20: communication bus, 21: control unit, 22: storage device, 23: input / output interface, 24: communication unit, 25: input operation unit, 26: display unit, 27: imaging unit, 28: microphone, 29: speaker, 30: communication bus, 31: control unit, 32: storage device, 33: position information acquisition unit, 34: communication unit, 35: input operation unit, 36: imaging unit, 37: PTZ mechanism, 40: communication bus, 41: control unit, 42: storage device, 43: input / output interface, 44: communication unit, 60: communication bus, 61: control unit, 62: storage device, 63: input / output interface, 64: communication unit, 65: input operation unit, 66: display unit, 67: imaging unit, 68: microphone, 69: speaker, 100: standby screen, 100a, 110a, 110b, 11c: video, 120: video display area, 130: operator display area, 200: customer service screen, 202: housing, 210: video display area, 212: store manual display area, 220: operator video display area, 223e: video, 230: store explanation material list area 240: Chat display area, 250: Camera button, 260: Home button, 263: Customer service end button, 265: Hold button, 310: Camera image display area, 320, 320a, 320b, 320c, 320d, 320e, 320f: Images, 340: Slider, 350: Timeline, K: Person, S: Attention area, U, Ue, Uf, Ug: Operator, W1, W2: Rectangular mark
Claims
1. A remote customer service device configured to support remote customer service and to capture a first image showing a customer; A plurality of cameras configured to capture a second image showing an area of interest; An information terminal operated by an operator who serves customers remotely; A remote customer service system that provides remote customer service by identifying at least one camera among the plurality of cameras associated with the remote customer service device; Displaying a first image captured by the remote customer service device and a second image captured by the specified camera on the information terminal; When any one of the second images is designated in response to an input operation on the information terminal, displaying the specified second image on the information terminal in an enlarged state together with the first image; Remote customer service system.
2. A plurality of remote customer service devices configured to support remote customer service and capture a first image showing a customer; A plurality of cameras configured to capture a second image showing an area of interest; An information terminal operated by an operator who serves customers remotely; A remote customer service system that provides remote customer service by displaying a plurality of first images captured by the plurality of remote customer service devices on the information terminal; accepting a designation of a first remote customer service device among the plurality of remote customer service devices in response to an input operation on the information terminal; identifying at least one camera among the plurality of cameras associated with the first remote customer service device; displaying, on the information terminal, a first image captured by the first remote customer service device and a second image captured by the specified camera; Remote customer service system.
3. A remote customer service device configured to support remote customer service and to capture a first image showing a customer; A plurality of cameras configured to capture a second image showing an area of interest; An information terminal operated by an operator who serves customers remotely; A remote customer service system that provides remote customer service by Identifying at least one camera associated with the remote customer service device among the plurality of cameras based on the identification information of the camera; displaying, on the information terminal, a first image captured by the remote customer service device and a second image captured by the specified camera; Remote customer service system.
4. A remote customer service device configured to support remote customer service and to capture a first image showing a customer; A plurality of cameras configured to capture a second image showing an area of interest; An information terminal operated by an operator who serves customers remotely; A remote customer service system that provides remote customer service by registering information indicating a correspondence relationship between the camera and the remote customer service device in response to an instruction from a user; Identifying at least one camera associated with the remote customer service device among the plurality of cameras based on the information indicating the correspondence relationship; displaying, on the information terminal, a first image captured by the remote customer service device and a second image captured by the specified camera; Remote customer service system.
5. A remote customer service device configured to support remote customer service and to capture a first image showing a customer; A plurality of cameras configured to capture a second image showing an area of interest; An information terminal operated by an operator who serves customers remotely; A remote customer service system that provides remote customer service by Identifying at least one camera associated with the remote customer service device among the plurality of cameras by analyzing the second video; displaying, on the information terminal, a first image captured by the remote customer service device and a second image captured by the specified camera; Remote customer service system.
6. A remote customer service device configured to support remote customer service and to capture a first image showing a customer; A plurality of cameras configured to capture a second image showing an area of interest; An information terminal operated by an operator who serves customers remotely; A remote customer service system that provides remote customer service by identifying at least one camera among the plurality of cameras associated with the remote customer service device; Displaying a first image captured by the remote customer service device and a second image captured by the specified camera on the information terminal; highlighting the remote customer service device shown in the second image; Remote customer service system.
7. A remote customer service device configured to support remote customer service and to capture a first image showing a customer; A plurality of cameras configured to capture a second image showing an area of interest; An information terminal operated by an operator who serves customers remotely; A remote customer service system that provides remote customer service by identifying at least one camera among the plurality of cameras associated with the remote customer service device; Displaying a first image captured by the remote customer service device and a second image captured by the specified camera on the information terminal; In response to an occurrence of a predetermined event in the second image, the first image corresponding to the second image in which the predetermined event occurred is highlighted. Remote customer service system.
8. A remote customer service device configured to support remote customer service and to capture a first image showing a customer; A plurality of cameras configured to capture a second image showing an area of interest; An information terminal operated by an operator who serves customers remotely; A remote customer service system that provides remote customer service by identifying at least one camera among the plurality of cameras associated with the remote customer service device; Displaying a first image captured by the remote customer service device and a second image captured by the specified camera on the information terminal; In response to occurrence of a predetermined event in the second image, the second image is superimposed and displayed as a wipe image on the first image corresponding to the second image in which the predetermined event occurred. Remote customer service system.
9. When a plurality of cameras are identified as cameras associated with the remote customer service device, displaying, on the information terminal, thumbnails of a plurality of second images captured by the identified plurality of cameras together with the first image; The remote customer service system according to any one of claims 1 to 8.
10. A first server capable of communicating with the remote customer service device and the information terminal; a second server capable of communicating with the plurality of cameras and the first server; Connected to the first video is provided from the remote customer service device to the information terminal via the first server, The second video is provided from the camera to the information terminal via the second server and the first server. The remote customer service system according to any one of claims 1 to 8.
11. In response to an instruction from a user, a location of the remote customer service device is identified; highlighting the area in which the remote customer service device is present; The remote customer service system according to claim 6.
12. By analyzing the second image, a presence area of the remote customer service device is identified, highlighting the area in which the remote customer service device is present; The remote customer service system according to claim 6.
13. Highlighting people present around the remote customer service device shown in the second image. The remote customer service system according to any one of claims 1 to 8.
14. In response to an instruction from a user, a surrounding area of the remote customer service device is identified; highlighting people present in the surrounding area; The remote customer service system according to claim 13.
15. By analyzing the second image, a person present in a surrounding area of the remote customer service device is detected, highlighting people present in the surrounding area; The remote customer service system according to claim 13.
16. In response to an occurrence of a predetermined event in the second video, a predetermined display process is executed in the information terminal. The remote customer service system according to any one of claims 1 to 8.
17. The predetermined event is that a determination condition related to at least one of the number of people present in the attention area of the second image and the duration of their stay is satisfied. The remote customer service system according to claim 16.
18. The predetermined display process is to highlight the second image in which the predetermined event has occurred. The remote customer service system according to claim 16.
19. The information terminal is configured to capture a third image showing the operator, The remote customer service system displays the third image captured by the information terminal on the remote customer service device. The remote customer service system according to any one of claims 1 to 8.
20. A remote customer service method executed by a remote customer service system, comprising: The remote customer service system includes: A plurality of remote customer service devices configured to support remote customer service and capture a first image showing a customer; A plurality of cameras configured to capture a second image showing an area of interest; An information terminal operated by an operator who serves customers remotely; We provide remote customer service by The remote customer service method includes: identifying at least one camera among the plurality of cameras associated with the remote customer service device; displaying, on the information terminal, a first image captured by the remote customer service device and a second image captured by the specified camera; When any one of the second images is designated in response to an input operation on the information terminal, displaying the specified second image on the information terminal in an enlarged state together with the first image; Including, Remote customer service methods.
21. A remote customer service method executed by a remote customer service system, comprising: The remote customer service system includes: A plurality of remote customer service devices configured to support remote customer service and capture a first image showing a customer; A plurality of cameras configured to capture a second image showing an area of interest; An information terminal operated by an operator who serves customers remotely; We provide remote customer service by The remote customer service method includes: displaying, on the information terminal, a plurality of first images captured by the plurality of remote customer service devices; receiving a designation of a first remote customer service device from among the plurality of remote customer service devices in response to an input operation on the information terminal; identifying at least one camera among the plurality of cameras associated with the first remote customer service device; displaying, on the information terminal, a first image captured by the first remote customer service device and a second image captured by the specified camera; Including, Remote customer service methods.
22. A remote customer service program for causing a remote customer service system to execute the remote customer service method according to claim 20 or 21.
Citation Information
Patent Citations
Supervisory system
JP1999164288A
Communication method of video telephone, and video telephone system
JP2006140652A
Photographing apparatus system and photographing device
JP2012039468A
Image distribution device
JP2013051678A
Information processing apparatus, information processing method, and information processing system
JP2015103104A
Cited By
Remote customer service system and remote customer service method
EP4835855A1
Remote customer service system and remote customer service method
JP2026177746A
