Remote customer service system, remote customer service method and remote customer service program

Abstract

To provide a remote customer service system capable of improving quality of remote customer service.SOLUTION: A remote customer service system 1 supports remote customer service, and provide remote customer service by a remote customer service device 2a configured to capture a first video showing customers, a plurality of cameras 3a-1 to 3a-10 configured to capture a second video showing a situation of a store, and an information terminal 6e operated by an operator Ue who provides remote customer service. The remote customer service system 1 identifies the cameras 3a-3 to 3a-8 associated with the remote customer service device 2a from among the plurality of cameras 3a-1 to 3a-10, and displays the first video captured by the remote customer service device 2a and the second video captured by the identified cameras 3a-3 to 3a-8 on the information terminal 6e.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure relates to a remote customer service system, a remote customer service method, and a remote customer service program. [Background technology]

[0002] Due in part to the impact of the COVID-19 pandemic, digital transformation (DX) is currently accelerating across most of society. DX is particularly prevalent in the customer service industry, which has been hit hard by labor shortages, and the introduction of remote customer service systems is currently attracting attention as a way to improve productivity and alleviate labor shortages.

[0003] In this regard, Patent Document 1 discloses a remote customer service system including a store terminal located in a store, an operator terminal, and a management device. In the remote customer service system disclosed in Patent Document 1, thumbnails of videos captured by multiple store terminals located in multiple stores are displayed side by side on a standby screen displayed on the operator terminal. When one of the thumbnails of the videos is designated by an operator, the store terminal that captures the video of the designated thumbnail is communicably connected to the operator terminal. Thereafter, a video of the operator captured by the operator terminal is displayed on the store terminal, and a video of the customer captured by the store terminal is displayed on the operator terminal. In this way, the operator can provide remote customer service to customers in the store from a location physically distant from the store, via the operator terminal and the store terminal. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2021-153294 Summary of the Invention [Problem to be solved by the invention]

[0005] The remote customer service system disclosed in Patent Document 1 allows a small number of operators to provide remote customer service at many stores, thereby significantly contributing to improving productivity and resolving labor shortages in the customer service industry. However, remote customer service using this remote customer service system has some drawbacks compared to conventional face-to-face customer service. In particular, while video captured by a store terminal can recognize customers within the store terminal's camera's field of view, it cannot recognize customers outside the store terminal's field of view. As a result, the operator cannot fully grasp the store's overall situation through the store terminal, and therefore cannot provide remote customer service that takes into account the store's surroundings. Furthermore, because the operator does not notice customers outside the field of view who would like to be served remotely, they miss opportunities to provide remote customer service to those customers. Thus, there is room for further improvement of remote customer service systems in light of the above points.

[0006] In view of the above, the present disclosure aims to provide a remote customer service system, a remote customer service method, and a remote customer service program that can improve the quality of remote customer service. [Means for solving the problem]

[0007] A remote customer service system according to one aspect of the present disclosure is a remote customer service system that supports remote customer service and provides remote customer service using a remote customer service device configured to capture a first video of a customer, multiple cameras configured to capture a second video of a predetermined area, and an information terminal operated by an operator providing remote customer service. The remote customer service system identifies at least one camera among the multiple cameras that is associated with the remote customer service device, and displays the first video captured by the remote customer service device and the second video captured by the identified camera on the information terminal.

[0008] According to the above configuration, the operator can provide remote customer service that takes into account the situation around the remote customer service device by checking both the first video and the second video displayed on the information terminal. In this regard, the operator can check the situation around the remote customer service device through the second video captured by the camera. On the other hand, in conventional remote customer service systems, only the first video captured by the remote customer service device is displayed on the information terminal, so the operator could only see a partial area in front of the remote customer service device. As a result, the operator was unable to fully grasp the overall situation through the information terminal. According to the remote customer service system of the present disclosure, it is possible to provide remote customer service that takes into account the overall situation around the remote customer service device through both the first video and the second video. Therefore, a remote customer service system that can improve the quality of remote customer service can be provided.

[0009] In addition, when multiple cameras are identified as cameras associated with the remote customer service device, the remote customer service system may display thumbnails of multiple second images taken by the identified multiple cameras on the information terminal along with the first image.

[0010] According to the above configuration, it is possible to provide remote customer service that takes into consideration the bird's-eye view of the surroundings of the remote customer service device through the first video and the second video displayed on the information terminal.

[0011] In addition, when one of the thumbnails of the plurality of second images is specified in response to an input operation on the information terminal, the remote customer service system may display on the information terminal an enlarged version of the second image corresponding to the specified thumbnail together with the first image.

[0012] According to the above configuration, the operator can check the detailed bird's-eye view of the situation around the remote customer service device through the enlarged second video image.

[0013] The remote customer service system may include a plurality of remote customer service devices. The remote customer service system may display, on the information terminal, a plurality of first images captured by the plurality of remote customer service devices, and, when a first remote customer service device is designated in response to an input operation on the information terminal, identify at least one camera associated with the first remote customer service device, and display, on the information terminal, the first image captured by the first remote customer service device and the second image captured by the identified camera.

[0014] According to the above configuration, when a first remote customer service device is selected from multiple remote customer service devices by an input operation, the operator can provide remote customer service through the first remote customer service device while taking into consideration the overall situation around the first remote customer service device, thereby improving the quality of remote customer service provided through the information terminal and the first remote customer service device.

[0015] The remote customer service system may be connected to a first server capable of communicating with the remote customer service device and the information terminal, and a second server capable of communicating with the cameras and the first server. The first video may be provided from the remote customer service device to the information terminal via the first server. The second video may be provided from the camera to the information terminal via the second server and the first server.

[0016] The remote customer service system may also identify the at least one camera associated with the remote customer service device based on the identification information of the camera.

[0017] In addition, the remote customer service system may register information indicating the correspondence between the camera and the remote customer service device in response to a user's instruction, and identify the at least one camera associated with the remote customer service device based on the information indicating the correspondence.

[0018] The remote customer service system may also identify the at least one camera associated with the remote customer service device by analyzing the second video.

[0019] The remote customer service system may also highlight the remote customer service device shown in the second image.

[0020] According to the above configuration, the operator can clearly recognize the remote customer service device present in the second image.

[0021] The remote customer service system may also specify an area where the remote customer service device exists, and highlight the area where the remote customer service device exists, in response to an instruction from a user.

[0022] The remote customer service system may also analyze the second video to identify an area where the remote customer service device is located and highlight the area where the remote customer service device is located.

[0023] The remote customer service system may also highlight people present around the remote customer service device shown in the second image.

[0024] According to the above configuration, the operator can clearly recognize people present around the remote customer service device through the second video.

[0025] The remote customer service system may also specify a surrounding area of ​​the remote customer service device in response to an instruction from a user, and highlight people present in the surrounding area.

[0026] The remote customer service system may also analyze the second video to detect people present in the surrounding area of ​​the remote customer service device and highlight the people present in the surrounding area.

[0027] Furthermore, the remote customer service system may execute a predetermined display process on the information terminal in response to an occurrence of a predetermined event in the second video.

[0028] According to the above configuration, the operator can clearly recognize that a predetermined event (for example, the area of ​​interest is congested) has occurred through a predetermined display process on the information terminal.

[0029] The predetermined event may be a determination condition related to at least one of the number of people present in the attention area of ​​the second video and the duration of their stay.

[0030] According to the above configuration, the occurrence of a predetermined event in the second video (for example, the area of ​​interest being crowded) is determined based on judgment conditions related to the number of people present in the area of ​​interest and the length of time they stay there, allowing the operator to provide remote customer service that takes into account the number of people present in the area of ​​interest and the length of time they stay there.

[0031] The predetermined display process may be to highlight the second video in which the predetermined event occurs.

[0032] According to the above configuration, the operator can clearly recognize through the highlighted second image that a predetermined event (for example, the area of ​​interest is congested) has occurred in the second image.

[0033] The predetermined display process may highlight a first image corresponding to a second image in which the predetermined event occurs.

[0034] According to the above configuration, the operator can clearly recognize through the highlighted first video that a predetermined event is occurring in a store or the like where the remote customer service device that captures the first video is installed. This allows the operator to understand the need for remote customer service in the store, thereby effectively preventing lost sales opportunities due to remote customer service. In this way, a remote customer service system that can improve the quality of remote customer service can be provided.

[0035] The predetermined display process may be to display the second video in which the predetermined event occurred together with a first video corresponding to the second video in which the predetermined event occurred.

[0036] According to the above configuration, the operator can clearly recognize the situation of a store or the like where a specific event (for example, the area of ​​interest is congested) is occurring through the first and second images, and can immediately understand the need for remote customer service in the store.

[0037] The information terminal may be configured to capture a third video showing the operator. The remote customer service system may display the third video captured by the information terminal on the remote customer service device.

[0038] A remote customer service system according to one aspect of the present disclosure is a remote customer service system that supports remote customer service and provides remote customer service using a plurality of remote customer service devices configured to capture a first video of a customer, a camera configured to capture a second video of a predetermined area, and an information terminal operated by an operator providing remote customer service. The remote customer service system identifies at least one remote customer service device among the plurality of remote customer service devices that is associated with the camera, and causes the second video captured by the camera and the first video captured by the identified remote customer service device to be displayed on the information terminal.

[0039] A remote customer service method according to one aspect of the present disclosure is executed by a remote customer service system. The remote customer service system supports remote customer service and provides remote customer service using a remote customer service device configured to capture a first video of a customer, multiple cameras configured to capture a second video of a predetermined area, and an information terminal operated by an operator providing remote customer service. The remote customer service method includes the steps of identifying at least one camera among the multiple cameras associated with the remote customer service device, and displaying the first video captured by the remote customer service device and the second video captured by the identified camera on the information terminal.

[0040] According to the above method, it is possible to provide remote customer service that takes into account the bird's-eye view of the surroundings of the remote customer service device through both the first video and the second video, thereby providing a remote customer service method that can improve the quality of remote customer service.

[0041] Also, a remote customer service program may be provided that causes a remote customer service system to execute the remote customer service method. [Effects of the Invention]

[0042] According to the present disclosure, it is possible to provide a remote customer service system, a remote customer service method, and a remote customer service program that can improve the quality of remote customer service. [Brief explanation of the drawings]

[0043] [Figure 1] 1 is a diagram showing a remote customer service system according to an embodiment of the present disclosure (hereinafter referred to as the present embodiment). [Figure 2] FIG. 2 illustrates an example of a hardware configuration of a remote customer service device. [Figure 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of a camera. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of a first server. [Figure 5] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information terminal. [Figure 6] 1 is a diagram for explaining the connection relationship between a plurality of remote customer service devices, a plurality of cameras, an information terminal, a first server, and a second server. [Figure 7] 1 is a flowchart (part 1) for explaining a series of processes executed by the remote customer service system according to the present embodiment. [Figure 8] 10 is a flowchart (part 2) for explaining a series of processes executed by the remote customer service system according to the present embodiment. [Figure 9] 10A is a diagram showing an example of an operator management table, FIG. 10B is a diagram showing an example of a company management table, and FIG. 10C is a diagram showing an example of a store management table. [Figure 10] FIG. 10 is a diagram showing an example of a standby screen on which thumbnails of multiple videos captured by multiple remote customer service devices are displayed. [Figure 11] FIG. 10 is a diagram showing an example of a customer service screen on which an image of a remote customer service device specified by an operator is displayed. [Figure 12] FIG. 10 is a diagram showing an example of a customer service screen on which an image of a remote customer service device specified by an operator and thumbnails of images from a plurality of cameras are displayed. [Figure 13] FIG. 10 is a diagram showing an example of a customer service screen on which an image from a camera specified by an operator is enlarged and displayed. [Figure 14] 10 is a flowchart illustrating a process of highlighting a thumbnail of an image of a remote customer service device in accordance with congestion in a set area of ​​interest in an image from a camera. [Figure 15] FIG. 10 is a diagram showing how one of the thumbnails of images from multiple remote customer service devices is highlighted in response to the congestion in the set attention area in the camera image. DETAILED DESCRIPTION OF THE INVENTION

[0044] (Configuration of remote customer service system 1) A remote customer service system 1 according to this embodiment will be described below with reference to the drawings. FIG. 1 is a diagram showing the remote customer service system 1 according to this embodiment. As shown in FIG. 1, the remote customer service system 1 includes remote customer service devices 2a-2c, cameras 3a-1-3c-10, a first server 4, a second server 5, and information terminals 6e-6g. These elements are connected to a communication network 7. The communication network 7 is configured by at least one of a local area network (LAN), a wide area network (WAN), the Internet, and a wireless core network. Note that the elements provided in the remote customer service system 1 are not limited to these elements, and other elements not shown in FIG. 1 may also be provided in the remote customer service system 1.

[0045] In the remote customer service system 1, operators Ue to Ug, who are physically located away from the stores, can remotely serve customers who visit their respective stores through information terminals 6e to 6g. For example, operator Ue can remotely serve customers who visit store A through information terminal 6e and a remote customer service device 2a installed in store A. Similarly, operator Uf can remotely serve customers who visit store B through information terminal 6f and a remote customer service device 2b installed in store B. Operator Ug can remotely serve customers who visit store C through information terminal 6g and a remote customer service device 2c installed in store C. Stores may be indoors or outdoors. The remote customer service system 1 may be installed not only in stores but also in various other locations, such as building reception desks, information centers, transportation counters, event venues, ticket booths, parking lots, warehouses, factories, guardhouses, and university campuses. Customers may receive various services, whether paid or free, at these various locations.

[0046] In the following description, for convenience of explanation, the remote customer service devices 2a to 2c may be collectively referred to simply as the remote customer service device 2. Similarly, the cameras 3a-1 to 3c-10 may be collectively referred to simply as the cameras 3, and the information terminals 6e to 6g may be collectively referred to simply as the information terminals 6. Furthermore, the operators Ue to Ug who operate the information terminals 6e to 6g may be collectively referred to simply as the operators U. Although three remote customer service devices 2, 30 cameras 3, and three information terminals 6 are shown in FIG. 1, the number of these devices is not particularly limited.

[0047] (Configuration of remote customer service device 2) Next, the hardware configuration of the remote customer service device 2 will be described below with reference to FIG. 2. In this embodiment, the remote customer service devices 2a to 2c have the same configuration. FIG. 2 is a diagram showing an example of the hardware configuration of the remote customer service device 2. The remote customer service devices 2a to 2c are communicably connected to a first server 4 via a communication network 7. The remote customer service devices 2 are configured to support remote customer service and are arranged in each store (e.g., a restaurant or a retail store) of a company to which the operators Ue to Ug who provide remote customer service belong. More specifically, the remote customer service device 2a is arranged in store A and is configured to support remote customer service for customers visiting store A. The remote customer service device 2b is arranged in store B and is configured to support remote customer service for customers visiting store B. The remote customer service device 2c is arranged in store C and is configured to support remote customer service for customers visiting store C. Customers can receive remote customer service from the operator U through the display unit 26 and speaker 29 of the remote customer service device 2.

[0048] 2, the remote customer service device 2 includes a control unit 21, a storage device 22, an input / output interface 23, a communication unit 24, an input operation unit 25, a display unit 26, an imaging unit 27, a microphone 28, and a speaker 29. These elements are connected to a communication bus 20.

[0049] The control unit 21 includes a memory and a processor. The memory is configured to store computer-readable instructions (programs). For example, the memory may include a read-only memory (ROM) storing various programs and a random access memory (RAM) having multiple work areas for storing various programs executed by the processor. The processor may include at least one of a central processing unit (CPU), a micro processing unit (MPU), and a graphics processing unit (GPU). The CPU may include multiple CPU cores. The GPU may include multiple GPU cores. The processor may be configured to load a specified program from various programs stored in the storage device 22 or the ROM onto the RAM and execute various processes in cooperation with the RAM.

[0050] The storage device 22 is a storage device (storage) such as a hard disk drive (HDD), a solid state drive (SSD), or a flash memory, and is configured to store programs and various data. The input / output interface 23 is an interface that enables connection between an external device and the remote customer service device 2, and includes an interface conforming to a predetermined communication standard such as the USB standard or the HDMI (registered trademark) standard. The communication unit 24 is configured to connect the remote customer service device 2 to the communication network 7. The communication unit 24 includes, for example, a wireless communication module and / or a wired communication module for wirelessly communicating with external devices such as a wireless base station or a wireless LAN router. The wireless communication module includes a transmitting / receiving antenna and a signal processing circuit. The wireless communication module may be a wireless communication module compatible with a short-range wireless communication standard such as Wi-Fi (registered trademark) or Bluetooth (registered trademark), or may be a wireless communication module compatible with an X-generation mobile communication system (e.g., a fourth-generation mobile communication system such as LTE) using a subscriber identity module (SIM).

[0051] The input operation unit 25 is, for example, a touch panel, a mouse, and / or a keyboard arranged over the video display of the display unit 26, and is configured to accept input operations by customers and generate operation signals in response to the input operations. The display unit 26 is, for example, configured by a video display and a video display circuit that drives and controls the video display.

[0052] The imaging unit 27 is configured to capture an image of the surrounding environment (particularly the front area) of the remote customer service device 2. In particular, the imaging unit 27 is configured to generate a video signal that indicates the surrounding environment of the remote customer service device 2, and includes an optical system, an image sensor, and an analog processing circuit. The optical system includes, for example, an optical lens and a color filter. The image sensor is configured by a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor) or the like. The analog processing circuit is configured to process the video signal (analog signal) photoelectrically converted by the image sensor, and includes, for example, an amplifier and an AD converter.

[0053] In this embodiment, the imaging unit 27 may be provided on the front surface of the housing 202 of the remote customer service device 2 (see FIG. 6). In this case, the imaging unit 27 can capture images of customers present in the area in front of the remote customer service device 2. In other words, the field of view of the imaging unit 27 is part of the area in front of the remote customer service device 2, so the remote customer service device 2 cannot capture images of the surrounding area excluding the area in front. The mounting position of the imaging unit 27 on the housing 202 is not particularly limited.

[0054] The microphone 28 is configured to collect voices uttered by customers and generate a voice signal (digital signal) based on the collected voices. The speaker 29 is configured to receive voice data from the control unit 21 and output voice to the outside based on the received voice data.

[0055] (Camera 3 configuration) Next, the hardware configuration of camera 3 will be described below with reference to FIG. 3. In this embodiment, cameras 3a-1 to 3c-10 have the same configuration. FIG. 3 is a diagram showing an example of the hardware configuration of camera 3. Camera 3 is configured to capture images showing its surrounding environment through photography, and may be placed inside or around a store (an example of a predetermined area). Camera 3 is a network-type surveillance camera that monitors the status of the store, and is communicatively connected to second server 5 via communication network 7.

[0056] As shown in FIG. 1, each of cameras 3a-1 to 3a-10 is fixedly installed at a predetermined location in store A and is configured to capture video images showing the bird's-eye view of store A through photography. Similarly, each of cameras 3b-1 to 3b-10 is fixedly installed at a predetermined location in store B and is configured to capture video images showing the bird's-eye view of store B through photography. Each of cameras 3c-1 to 3c-10 is fixedly installed at a predetermined location in store C and is configured to capture video images showing the bird's-eye view of store C through photography. Each camera 3 transmits video data to second server 5 via communication network 7. Second server 5 stores the video data transmitted by each camera 3.

[0057] 3, the camera 3 includes a control unit 31, a storage device 32, a position information acquisition unit 33, a communication unit 34, an input operation unit 35, an imaging unit 36, and a PTZ mechanism 37, and these elements are connected to a communication bus 30. The camera 3 may also include a built-in battery (not shown). The camera 3 may also include a microphone and a speaker.

[0058] The control unit 31 includes a memory and a processor. The memory is configured to store computer-readable instructions (programs). For example, the memory is composed of a ROM and a RAM. The processor is configured, for example, by at least one of a CPU, an MPU, and a GPU. The storage device 32 is, for example, a storage device such as an HDD, an SSD, or a flash memory, and is configured to store programs and various data. The position information acquisition unit 33 is configured to acquire position information (longitude, latitude) of the camera 3, and is, for example, a GPS (Global Positioning System) receiver.

[0059] The communication unit 34 is configured to connect the camera 3 to the communication network 7. The communication unit 34 includes, for example, a wireless communication module and / or a wired communication module for wireless communication with external devices such as a wireless base station or a wireless LAN router. The input operation unit 35 is configured to accept input operations by the operator and to generate operation signals in accordance with the input operations by the operator. The imaging unit 36 ​​is configured to capture images of the environment surrounding the camera 3. In particular, the imaging unit 36 ​​is configured to generate video signals indicative of the environment surrounding the camera 3, and includes an optical system, an image sensor, and an analog processing circuit.

[0060] The PTZ mechanism 37 includes a pan mechanism, a tilt mechanism, and a zoom mechanism. The pan mechanism is configured to change the orientation of the camera 3 in the horizontal direction. The tilt mechanism is configured to change the orientation of the camera 3 in the vertical direction. The zoom mechanism is configured to change the angle of view of the camera 3 to enlarge (zoom in) or reduce (zoom out) an image showing an object to be captured. The zoom mechanism may change the angle of view of the camera 3 optically by changing the focal length of an optical lens included in the imaging unit 36, or may change the angle of view of the camera 3 digitally. In this embodiment, in response to an input operation by the operator U on the information terminal 6, an instruction signal instructing the camera 3 to pan, tilt, and / or tilt is transmitted from the information terminal 6 to the camera 3 via the second server 5. In this case, the control unit 31 drives the PTZ mechanism 37 in response to the received instruction signal, thereby realizing the pan, tilt, and zoom functions (PTZ functions) of the camera 3 in real time. In this manner, the PTZ function of the camera 3 can be realized through remote control of the information terminal 6.

[0061] The camera 3 can transmit a video (video stream) showing the surrounding environment of the camera 3 to the second server 5 via the communication network 7 in real time.

[0062] (Configuration of first server 4) Next, the hardware configuration of the first server 4 will be described below with reference to FIG. 4. FIG. 4 is a diagram showing an example of the hardware configuration of the first server 4. In this embodiment, the first server 4 may be configured as a web server, an application server, and a video data server. In other words, the first server 4 has a function of providing web pages, a function of providing remote customer service applications, and a function of providing video data.

[0063] In this regard, the first server 4 is configured to transmit data (e.g., HTML files, CSS files, image / video files, program files, etc.) for displaying the standby screen 100 (see FIG. 10) and the customer service screen 200 (see FIG. 11) on the web browser of the information terminal 6. In this manner, the first server 4 functions as a server for providing SaaS (System as a Service). The first server 4 may be configured with multiple servers. The first server 4 may be constructed on-premise or as a cloud server. The first server 4 is communicably connected to the remote customer service devices 2a-2c and the information terminals 6e-6g via a communication network 7. The first server 4 is configured to transmit video data captured by the remote customer service devices 2a-2c to each of the information terminals 6e-6g.

[0064] As shown in FIG. 4, the first server 4 includes a control unit 41, a storage device 42, an input / output interface 43, and a communication unit 44. These elements are connected to a communication bus 40. The control unit 41 includes a memory and a processor. The memory is configured to store computer-readable instructions. In particular, the memory may store a program for causing the processor to execute a series of processes executed by the first server 4. The memory includes a ROM and a RAM. The processor includes at least one of a CPU, an MPU, and a GPU.

[0065] The storage device 42 is, for example, a storage device (storage) such as an HDD, SSD, or flash memory, and is configured to store programs and various data. The storage device 42 stores an operator management table, a company management table, and a store management table (see FIG. 9), which will be described later. The storage device 42 may also store images captured by the remote customer service devices 2a to 2c and / or images captured by the information terminals 6e to 6g. The input / output interface 43 is an interface that enables connection between an external device and the first server 4. The communication unit 44 may include various wired communication modules for communicating with external devices on the communication network 7.

[0066] (Configuration of the second server 5) The second server 5 may have the same hardware configuration as the first server 4 shown in FIG. 4. Like the first server 4, the second server 5 includes a control unit, a storage device, an input / output interface, and a communication unit, all of which are connected via a communication bus. The second server 5 is communicatively connected to each of the cameras 3b-1 to 3b-10 via a communication network 7. The second server 5 is configured to receive video images captured by the cameras 3 and then store and manage the received video images. The second server 5 may store a camera management table for managing the cameras 3 and a video management table for managing the video images captured by the cameras 3. The camera management table includes information about the cameras 3 installed in each store. The video management table includes information about the video data of each camera 3. The second server 5 is communicatively connected to the first server 4 via an API (Application Programming Interface). In other words, the second server 5 is configured to transmit desired video images to the first server 4 in response to a video transmission request from the first server 4.

[0067] (Configuration of information terminal 6) Next, the hardware configuration of the information terminal 6 will be described below with reference to FIG. 5. In this embodiment, the information terminals 6e to 6g have the same configuration. FIG. 5 is a diagram showing an example of the hardware configuration of the information terminal 6. The information terminal 6 is operated by an operator U who provides remote customer service. The information terminal 6 is configured to capture video of the operator U and acquire the voice of the operator U. The information terminal 6 is located in a position physically separated from the remote customer service device 2. In this embodiment, the information terminal 6e is operated by the operator Ue, the information terminal 6f is operated by the operator Uf, and the information terminal 6g is operated by the operator Ug.

[0068] The information terminal 6 may be, for example, a personal computer, a smartphone, a tablet, or a wearable device (e.g., AR / VR / MR glasses, etc.) worn by the operator U. The information terminal 6 has a web browser. The remote customer service application provided by the first server 4 may run on the web browser of the information terminal 6. Furthermore, the remote customer service application may be downloaded and then installed on the information terminal 6.

[0069] 5, the information terminal 6 includes a control unit 61, a storage device 62, an input / output interface 63, a communication unit 64, an input operation unit 65, a display unit 66, an imaging unit 67, a microphone 68, and a speaker 69. These elements are connected to a communication bus 60.

[0070] The control unit 61 includes a memory and a processor. The memory is configured to store computer-readable instructions. In particular, the memory may store a program for causing the processor to execute a series of processes executed by the information terminal 6. The memory is configured from a ROM and a RAM. The processor is configured from at least one of a CPU, an MPU, and a GPU. The storage device 62 is, for example, a storage device such as an HDD, an SSD, or a flash memory, and is configured to store programs and various data.

[0071] The input / output interface 63 is an interface that enables connection between an external device and the information terminal 6. The communication unit 64 is configured to connect the information terminal 6 to the communication network 7. The communication unit 64 includes, for example, a wireless communication module and / or a wired communication module. The input operation unit 65 is, for example, a touch panel, a mouse, and / or a keyboard that are arranged over the video display of the display unit 66, and is configured to accept input operations by the operator U and generate operation signals in response to the input operations. The display unit 66 is, for example, configured by a video display and a video display circuit that drives and controls the video display.

[0072] The imaging unit 67 is configured to generate a video signal indicating the surrounding environment of the information terminal 6 (particularly, a video signal indicating the operator U), and includes an optical system, an image sensor, and an analog processing circuit. The microphone 68 is configured to collect the voice of the operator U and generate a voice signal based on the collected voice. The speaker 69 is configured to receive voice data from the control unit 61 and output voice to the outside based on the received voice data. In this embodiment, after the voice data of the operator U acquired by the microphone 68 is transmitted from the information terminal 6 to the remote customer service device 2 via the communication network 7, the voice of the operator U is output from the speaker 29 of the remote customer service device 2. Similarly, after the voice data of the customer acquired by the microphone 28 is transmitted from the remote customer service device 2 to the information terminal 6 via the communication network 7, the voice of the customer is output from the speaker 69 of the information terminal 6.

[0073] (Remote Customer Service Method According to the Present Embodiment: Flow of a Series of Processes Executed by the Remote Customer Service System 1) Next, a series of processes (remote customer service method according to this embodiment) executed by the remote customer service system 1 according to this embodiment will be described below mainly with reference to Fig. 6 to Fig. 8. Fig. 6 is a diagram for explaining the connection relationship between the remote customer service devices 2a to 2c, the cameras 3a-1 to 3a-10, the information terminal 6e, the first server 4, and the second server 5. Figs. 7 and 8 are flowcharts for explaining a series of processes executed by the remote customer service system 1.

[0074] In this example, an operator Ue operating an information terminal 6e (an example of a first information terminal) specifies a remote customer service device 2a (an example of a first remote customer service device) and remotely serves a customer in a store A. In this case, images from cameras 3a-3 to 3a-8 associated with the remote customer service device 2a, among cameras 3a-1 to 3a-10 arranged in the store A, are displayed on the information terminal 6e. As shown in FIG. 6, a first server 4 is communicably connected to the information terminal 6e and the remote customer service devices 2a to 2c. The first server 4 is communicably connected to a second server 5 via API linkage. The second server 5 is communicably connected to the cameras 3a-1 to 3a-10.

[0075] As shown in FIG. 7, in step S1, the information terminal 6e receives the login information of the operator Ue for logging in to the remote customer service application through an input operation by the operator Ue, and then transmits the login information to the first server 4. The first server 4 performs login authentication of the operator Ue based on the login information of the operator Ue and the operator management table (see FIG. 9(a)). Furthermore, the first server 4 identifies the company ID of the company to which the operator Ue belongs by referring to the operator management table, and then identifies the store ID of each store corresponding to the company ID by referring to the company management table (see FIG. 9(b)). Thereafter, the first server 4 identifies the remote customer service device ID of the remote customer service device 2 corresponding to each store ID by referring to the store management table (see FIG. 9(c)). In this way, the first server 4 identifies the remote customer service devices 2a to 2c installed in the stores A to C managed by the operator Ue (step S2). Note that the term "company" here refers not only to the company itself but also to a broad concept including the brands owned by the company. Furthermore, the term "store" as used here refers not only to the store itself but also to a broader concept that includes the area in which the store is located. Furthermore, a concept of "room" may be established for each company, and multiple stores may be associated and managed with each room. For example, in the case of a company operating multiple stores (or multiple business types), dividing rooms by jurisdiction (or jurisdiction type) would improve convenience. In this case, the correspondence between company IDs and room IDs is stored in advance as a room table (not shown), and the company ID in Figure 9(b) is replaced with the room ID. After identifying the company ID, a list of corresponding rooms is displayed on the information terminal (not shown), and the operator can identify a room ID from the list based on their selection, and then identify the corresponding store ID and remote customer service device ID.

[0076] Next, the first server 4 transmits a video request to the remote customer service devices 2a to 2c (step S3). In response to the video request transmitted from the first server 4, the remote customer service devices 2a to 2c transmit a video (an example of a first video) showing the surrounding environment of the remote customer service device 2 to the first server 4 (step S4). In step S5, the first server 4 receives the video (video stream) captured by the remote customer service devices 2a to 2c and transmits data (e.g., HTML file, CSS file, image / video file, program file, etc.) for displaying a standby screen 100 (see FIG. 10) to the information terminal 6e. Thereafter, the standby screen 100 is displayed on the display unit 66 of the information terminal 6e (step S6). In particular, the standby screen 100 may be displayed on a web browser of the information terminal 6e.

[0077] 10, the standby screen 100 has a video display area 120 and an operator display area 130. Thumbnails of videos 110a to 110c captured by the remote customer service devices 2a to 2c are displayed in the video display area 120. Information on operators Ue to Ug who are logged in to the remote customer service application and who correspond to the same company ID is displayed in the operator display area 130.

[0078] Next, in step S7, the first server 4 detects person K in the video 110a captured by the remote customer service device 2a using an image recognition model or the like, and then transmits a highlighting instruction to the information terminal 6e to highlight a thumbnail of the video 110a. Thereafter, in step S8, the information terminal 6e highlights the thumbnail of the video 110a captured by the remote customer service device 2a in response to the highlighting instruction transmitted from the first server 4 (see FIG. 10). For example, the visual highlighting of the thumbnail of the video 110a may be achieved by making the outer frame of the thumbnail of the video 110a thicker and in a more noticeable color.

[0079] In the process of step S6, only the standby screen 100 is displayed on the display unit 66 of the information terminal 6e. However, if the operators Uf and Ug are also logged in to the remote customer service application, the standby screen 100 is also displayed on the display units 66 of the information terminals 6e and 6f. That is, the first server 4 transmits data for displaying the standby screen 100 to each of the information terminals 6e to 6g. Similarly, in the process of step S7, a highlighting instruction for highlighting the thumbnail of the video 110a is transmitted to each of the information terminals 6e to 6g. Thereafter, in step S8, each of the information terminals 6e to 6g highlights the thumbnail of the video 110a in response to the highlighting instruction.

[0080] Next, in step S9, when operator Ue wishes to provide remote customer service at store A, operator Ue specifies the thumbnail of video 110a displayed on standby screen 100 via input operation unit 65. Specifically, as shown in FIG. 10, the thumbnail of video 110a is double-clicked. At this time, information terminal 6e accepts the designation of remote customer service device 2a in response to operator Ue's input operation, and transmits the designation information of remote customer service device 2a to first server 4 (step S10). Next, the first server 4 transmits data (e.g., HTML file, CSS file, image / video file, program file, etc.) for displaying customer service screen 200 (see FIG. 11) of remote customer service device 2a in response to the designation information (step S11). Thereafter, customer service screen 200 is displayed on display unit 66 of information terminal 6e (step S12).

[0081] As shown in FIG. 11 , the customer service screen 200 includes a video display area 210, a store manual display area 212, an operator video display area 220, a store explanatory material list area 230, and a chat display area 240. The video display area 210 displays a video 110a captured by the remote customer service device 2a. The store manual display area 212 displays the customer service manual for store A. The operator video display area 220 displays a video 223e of the operator Ue captured by the imaging unit 67 of the information terminal 6e. The store explanatory material list area 230 displays a list of explanatory materials for store A, from which content to be displayed on the display unit of the remote customer service device can be selected and displayed according to the operator's instructions. The chat display area 240 displays the contents of a chat between the operator Ue and a customer at store A.

[0082] The customer service screen 200 also has a home button 260, a camera button 250, a hold button 265, and an end customer service button 263. When the home button 260 is operated by the operator Ue, the customer service screen 200 shown in FIG. 11 is displayed. When the camera button 250 is operated by the operator Ue, the customer service screen 200 shown in FIG. 12 is displayed. When the hold button 265 is operated by the operator Ue, remote customer service for a customer in store A is put on hold. In this case, the display of the video showing the operator Ue on the remote customer service device 2a may be interrupted. When the end customer service button 263 is operated by the operator Ue, the screen displayed on the information terminal 6e switches from the customer service screen 200 to the standby screen 100 shown in FIG. 10.

[0083] Next, in step S13, the information terminal 6e transmits a video 223e (an example of a third video) captured by the information terminal 6e and showing the operator Ue to the remote customer service device 2a. Thereafter, the video 223e (video stream) captured by the information terminal 6e is displayed on the display unit 26 of the remote customer service device 2a (step S14). In this way, when the operator Ue provides remote customer service in the store A, the first server 4 starts communication between the information terminal 6e and the remote customer service device 2a, so that the video 110a captured by the remote customer service device 2a, including the customer, is displayed on the information terminal 6e, and the video 223e captured by the information terminal 6e, including the operator, is displayed on the remote customer service device 2a. In this case, the remote customer service device 2a may transmit the video 110a to the information terminal 6e via the first server 4, or may transmit the video 110a directly to the information terminal 6e via P2P (Peer to Peer) without going through the first server 4. Similarly, the information terminal 6e may transmit the video 223e to the remote customer service device 2a via the first server 4, or may transmit the video 223e directly to the remote customer service device 2a by P2P without going through the first server 4.

[0084] Similarly to the video data, audio data is also transmitted and received between the information terminal 6e and the remote customer service device 2a. That is, the customer's audio data acquired by the microphone 28 of the remote customer service device 2a is output from the speaker 69 of the information terminal 6e, and the operator Ue's audio data acquired by the microphone 68 of the information terminal 6e is output from the speaker 29 of the remote customer service device 2a.

[0085] Next, in step S15, the information terminal 6e transmits a request to display the camera image in response to an input operation by the operator Ue to the first server 4. Specifically, as shown in FIG. 11 , the operator Ue operates a camera button 250 provided on the customer service screen 200, whereby the request to display the camera image is transmitted from the information terminal 6e to the first server 4.

[0086] In step S16, in response to the request to display the camera footage, the first server 4 transmits a request to the second server 5 to send the camera footage associated with the remote customer service device 2a. Here, the request to send the camera footage may include information about the cameras 3a-1 to 3a-10 installed in store A (e.g., the camera IDs of the cameras 3a-1 to 3a-10). In this case, the first server 4 refers to the store management table (FIG. 9(c)) to obtain information about the cameras 3a-1 to 3a-10 installed in store A, and then transmits a request to send the camera footage including the information about the cameras 3a-1 to 3a-10 installed in store A to the second server 5.

[0087] In step S17, the second server 5 identifies the cameras 3a-3 to 3a-8 associated with the remote customer service device 2a in response to the camera video transmission request transmitted from the first server 4. Here, the cameras 3 associated with the remote customer service device 2a may be defined as cameras 3 arranged in the vicinity of the remote customer service device 2a and / or cameras 3 that capture video including the presence of the remote customer service device 2a.

[0088] (First example of a process for identifying cameras 3a-3 to 3a-8 associated with remote customer service device 2a) In the first example, the remote customer service device 2a is automatically associated with the cameras 3a-3 to 3a-8 based on the camera names. Specifically, the second server 5 may identify the names of the cameras 3a-1 to 3a-10 based on the camera IDs of the cameras 3a-1 to 3a-10 included in the camera video transmission request and on data indicating the correspondence between the camera IDs and the camera names (an example of camera identification information). Next, the second server 5 may identify the cameras 3a-3 to 3a-8 associated with the remote customer service device 2a based on the names of the cameras 3a-1 to 3a-10. For example, if the names of the cameras 3a-3 to 3a-8 include information about the remote customer service device 2a (e.g., a character string indicating the remote customer service device 2a), the cameras 3a-3 to 3a-8 may be identified as the cameras associated with the remote customer service device 2a. The data indicating the correspondence between the camera IDs and the camera names (an example of camera identification information) may be registered in advance by the operator U. In this example, the camera 3 associated with the remote customer service device 2 is identified based on the camera name as an example of the identification information of the camera 3, but the identification information of the camera 3 is not limited to the camera name.

[0089] (Second Example of Processing for Identifying Cameras 3a-3 to 3a-8 Associated with Remote Customer Service Device 2a) In the second example, the remote customer service device 2a and the cameras 3a-3 to 3a-8 are manually associated with each other based on a user instruction. It is assumed that this user instruction is mainly given from the information terminal 6 of the operator U, but it may also be given from an information processing device (not shown) of the company that manages them. Specifically, the request to send the camera footage may include information about the remote customer service device 2a (e.g., the ID of the remote customer service device 2a). In this case, the second server 5 may identify the cameras 3a-3 to 3a-8 associated with the remote customer service device 2a based on the ID of the remote customer service device 2a and data indicating the correspondence between the multiple remote customer service devices 2 and the multiple cameras 3, which are included in the request to send the camera footage.

[0090] The data indicating the correspondence between the multiple remote customer service devices 2 and the multiple cameras 3 may indicate the correspondence between the IDs of the multiple remote customer service devices 2 and the multiple camera IDs. The data indicating the correspondence may also include information about the multiple cameras 3 associated with each remote customer service device 2. For example, the data indicating the correspondence may indicate the correspondence between the ID of the remote customer service device 2a and the IDs of the cameras 3a-3 to 3a-8. In this way, it is possible to identify the cameras 3a-3 to 3a-8 associated with the remote customer service device 2a based on the data indicating the correspondence. The data indicating the correspondence between the multiple remote customer service devices 2 and the multiple cameras 3 may also be registered in advance by the operator U.

[0091] (Third Example of Processing for Identifying Cameras 3a-3 to 3a-8 Associated with Remote Customer Service Device 2a) In the third example, the remote customer service device 2a is automatically associated with the cameras 3a-3 to 3a-8 based on video analysis. Specifically, the camera video transmission request may include information about the cameras 3a-1 to 3a-10 installed in the store A (e.g., the camera IDs of the cameras 3a-1 to 3a-10). In this case, the second server 5 acquires the videos captured by the cameras 3a-1 to 3a-10 based on the camera IDs of the cameras 3a-1 to 3a-10 included in the camera video transmission request. The second server 5 then analyzes the videos captured by the cameras 3a-1 to 3a-10 to extract videos showing the remote customer service device 2a from the videos captured by the cameras 3a-1 to 3a-10. The second server 5 then identifies the cameras 3a-3 to 3a-8 capturing videos showing the presence of the remote customer service device 2a as the cameras associated with the remote customer service device 2a. When extracting the video image showing the remote customer service device 2a, the second server 5 may use an image recognition model, which is a trained model constructed by machine learning. In this case, the image recognition model may be constructed from training data including image data and information about the remote customer service device 2a included in the image data.

[0092] Next, in step S18, the second server 5 transmits the images (thumbnail display) of the cameras 3a-3 to 3a-8 associated with the remote customer service device 2a to the first server 4. Next, the first server 4 transmits the images of the cameras 3a-3 to 3a-8 to the information terminal 6e (step S19). Thereafter, as shown in FIG. 12, the information terminal 6e displays thumbnails of the images 320a to 320f (examples of the second images) captured by the cameras 3a-3 to 3a-8 on the customer service screen 200 (step S20). Note that if only one camera is identified in step S17, FIG. 12 may be omitted and FIG. 13 may be displayed instead. In that case, steps S18 to S22 may be omitted, and the image of the identified camera may be enlarged and displayed in the camera image display area 310.

[0093] The customer service screen 200 shown in FIG. 12 includes a video display area 210, a store manual display area 212, and a camera video display area 310. In the camera video display area 310, thumbnails of videos 320a to 320f captured by cameras 3a-3 to 3a-8 are displayed side by side. Here, video 320a is the video captured by camera 3a-3. Video 320f is the video captured by camera 3a-8. In this way, on the customer service screen 200 shown in FIG. 12, video 110a captured by the remote customer service device 2a and thumbnails of videos 320a to 320f captured by cameras 3a-3 to 3a-8 associated with the remote customer service device 2a are simultaneously displayed on the same screen.

[0094] As shown in Fig. 12, the remote customer service device 2a and a person K present in the vicinity of the remote customer service device 2a are highlighted in the images 320a to 320f. For example, in the image 320b captured by the camera 3a-4, a rectangular mark W1 surrounding the presence area of ​​the remote customer service device 2a is displayed so that the presence area of ​​the remote customer service device 2a is highlighted. Similarly, a rectangular mark W2 surrounding the person K present near the remote customer service device 2a is displayed in the image 320b so that the person K is highlighted. Note that the rectangular mark W2 may be displayed simply when the person K is detected in the image, and may be further highlighted in an attention color or the like when the person K enters the vicinity of the remote customer service device 2a.

[0095] (Display process of rectangular mark W1 surrounding the area where the remote customer service device 2a is located in the camera image) For example, the presence area of ​​the remote customer service device 2a may be set in response to an input operation by the operator U on the information terminal 6. While such user instructions are assumed to be mainly given from the information terminal 6 of the operator U, they may also be given from an information processing device (not shown) of a company that manages them. For example, when the operator U specifies the presence area of ​​the remote customer service device 2a in the video 320b while the video 320b is displayed on the information terminal 6, information indicating the presence area of ​​the remote customer service device 2a in the video 320b (e.g., coordinate information defining the presence area in the video 320b) is transmitted from the information terminal 6 to the second server 5. Thereafter, the second server 5 identifies the presence area of ​​the remote customer service device 2a based on the information indicating the presence area, and displays a rectangular mark W1 surrounding the presence area of ​​the remote customer service device 2a in the video 320b. Thereafter, a thumbnail of the video 320b with the rectangular mark W1 displayed thereon is displayed on the customer service screen 200. Furthermore, the rectangular mark W1 is set and displayed in each of the videos 320a, 320d, and 320e in the same manner as above. Furthermore, the second server 5 may analyze the video 320b using an image recognition model to identify the presence area of ​​the remote customer service device 2a shown in the video 320b, and then display a rectangular mark W1 surrounding the identified presence area of ​​the remote customer service device 2a.

[0096] (Display process of rectangular mark W2 surrounding person K in camera image) The peripheral area of ​​the remote customer service device 2a may also be set in response to an input operation by the operator U on the information terminal 6. While such user instructions are primarily assumed to be given from the operator U's information terminal 6, they may also be given from an information processing device (not shown) of a company that oversees the information terminal 6. For example, when the operator U specifies the peripheral area of ​​the remote customer service device 2a in the video 320b while the video 320b is displayed on the information terminal 6, information indicating the peripheral area of ​​the remote customer service device 2a in the video 320b (e.g., coordinate information defining the peripheral area in the video 320b) is transmitted from the information terminal 6 to the second server 5. The peripheral area may be set automatically or manually to an area within a predetermined distance from the remote customer service device 2a or an area where people have been lingering or the like with a predetermined frequency or more in the past. The second server 5 then sets the peripheral area in the video 320b based on the information transmitted from the information terminal 6. Thereafter, when the second server 5 detects that person K is present in the peripheral area, it displays a rectangular mark W2 surrounding person K on the video 320b to highlight person K. Then, a thumbnail of the video 320b displaying the rectangular mark W2 is displayed on the customer service screen 200. Alternatively, the rectangular mark W2 may be displayed simply when person K is detected in the video, and then the rectangular mark W2 may be highlighted using a prominent color or the like when person K's presence in the peripheral area is detected. Furthermore, using a similar method to the above, the peripheral area of ​​the remote customer service device 2a is set in each of the videos 320a, 320d, and 320e, and the rectangular mark W2 surrounding person K present in the set peripheral area is displayed. The second server 5 may use an image recognition model when detecting that person K is present in the peripheral area. In this case, the image recognition model may be constructed using learning data consisting of image data and information about the person contained in the image data. In particular, the second server 5 may analyze the video 320b using an image recognition model to detect a person present within the surrounding area of ​​the remote customer service device 2a, and then display a rectangular mark W2 surrounding the detected person on the video 320b.

[0097] In this way, because the rectangular marks W1 and W2 are displayed in some of the images 320a to 320f, the operator Ue can clearly recognize the remote customer service device 2a present in each image by the rectangular mark W1. Similarly, the operator Ue can clearly recognize the person K present around the remote customer service device 2a shown in each image by the rectangular mark W2. Therefore, the operator Ue can visually recognize the remote customer service device 2a and the person K in its vicinity, and can instantly grasp the situation around the remote customer service device 2a. Note that in this embodiment, the remote customer service device 2a and the person K are highlighted in the image from the camera 3 by the rectangular marks W1 and W2, but the visual manner in which the remote customer service device 2a and the person K are highlighted is not limited to the rectangular marks W1 and W2.

[0098] According to this embodiment, the operator Ue can provide remote customer service that takes into account the store conditions around the remote customer service device 2a by checking both the video 110a (an example of a first video) and thumbnails of the videos 320a to 320f (an example of a second video) displayed on the information terminal 6e. In this respect, the operator Ue can check the store conditions around the remote customer service device 2a through the videos 320a to 320f. In contrast, in the past, only the video captured by the remote customer service device 2a was displayed on the information terminal 6e, so the operator Ue could only see a partial area in front of the remote customer service device 2a. For this reason, the operator Ue was unable to fully grasp the store conditions from a bird's-eye view through the information terminal 6e. According to the remote customer service system 1 of the present disclosure, it is possible to provide remote customer service that takes into account the store conditions around the remote customer service device 2a through both the video 110a and the videos 320a to 320f. Therefore, a remote customer service system 1 that can improve the quality of remote customer service can be provided.

[0099] 8, in step S21, information terminal 6e receives a designation of a thumbnail of video 320b from camera 3a-4 in response to an input operation by operator Ue, and then transmits a request for enlarged display of video 320b to first server 4. For example, as shown in FIG. 12, the thumbnail of video 320b is double-clicked.

[0100] In step S22, the first server 4 transmits a request to the second server 5 to send the enlarged image 320b of the camera 3a-4 in response to the enlarged image request from the information terminal 6e. In step S23, the second server 5 transmits the enlarged image 320b to the first server 4 in response to the transmission request from the first server 4. The first server 4 then transmits the enlarged image 320b to the information terminal 6e (step S24). As a result, as shown on the customer service screen 200 in FIG. 13, the enlarged image 320b is displayed on the information terminal 6e (step S25). Note that the enlarged image here refers to the display of the original image relative to the thumbnail of the image 320b, but may also refer to an image in which a portion of the original image (for example, an area including the remote customer service device 2a and the customer) is zoomed in.

[0101] 13, a viewer for video 320b is displayed in the camera video display area 310. The viewer for video 320b includes an enlarged version of video 320b, a timeline 350 indicating the playback time of video 320b, and a slider 340 that can be slid on the timeline 350. The operator Ue can change the playback time of video 320b by moving the slider 340 on the timeline 350 of the viewer for video 320b. For example, if the operator Ue moves the slider 340 to the 10:30 position, the information terminal 6e transmits a transmission request for video 320b in a time slot around 10:30 to the first server 4. In response to the transmission request, the first server 4 acquires video 320b in a time slot around 10:30 from the second server 5 and transmits the acquired video 320b to the information terminal 6e. In this way, the video 320b from the time period around 10:30 is displayed in the camera video display area 310.

[0102] In this way, the operator Ue can check in detail the store situation around the remote customer service device 2a (especially the state of people near the remote customer service device 2a) through both the image 100a and the enlarged image 320b. Therefore, it is possible to provide remote customer service that takes into account the store situation around the remote customer service device 2a, and a remote customer service system 1 that can improve the quality of remote customer service can be provided.

[0103] (Remote Customer Service Method According to a Modification of the Present Embodiment: Processing for Highlighting Thumbnail of Video 110a Depending on Congestion in the Area of ​​Attention) Next, mainly with reference to Fig. 14 and Fig. 15, a process (a remote customer service method according to a modified example of this embodiment) for highlighting a thumbnail of the video 110a of the remote customer service device 2a in accordance with congestion in the attention area S (see Fig. 15) set in the video 320a captured by the camera 3a-3 will be described below. Fig. 14 is a flowchart for explaining a process for highlighting a thumbnail of the video 110a of the remote customer service device 2a in accordance with congestion in the attention area S set in the video 320a of the camera 3a-3. Fig. 15 is a diagram showing how the video 110a is highlighted among the thumbnails of the videos 110a-110c of the remote customer service devices 2a-2c in accordance with congestion in the attention area S set in the video 320a of the camera 3a-3.

[0104] 7, in steps S7 and S8, the thumbnail of the video 110a is highlighted in response to the detection of a person K in the video 110a. On the other hand, in this modified example, even if a person K is not detected in the video 110a, the thumbnail of the video 110a is highlighted, which is different from the processing in steps S7 and S8. In this respect, this modified example differs from steps S7 and S8 in that the thumbnail of the video 110a is highlighted in response to the status of the attention area S set in the video 320a of the camera 3a-3 associated with the remote customer service device 2a.

[0105] 14, steps S31 to S34 are the same as steps S1 to S4 in Fig. 7, and therefore description thereof will be omitted. In step S35, the first server 4 receives the images captured by the remote customer service devices 2a to 2c, and then transmits data for displaying standby screen 100 (see Fig. 15) to information terminal 6e. Thereafter, standby screen 100 shown in Fig. 15 is displayed on display unit 66 of information terminal 6e (step S36).

[0106] 15, standby screen 100 has a video display area 120 and an operator display area 130. Thumbnails of videos 110a to 110c captured by remote customer service devices 2a to 2c are displayed in video display area 120. At this point, the outer frame of the thumbnail of video 110a is not highlighted, and video 320a is not superimposed on video 110a as a wipe video.

[0107] Next, in step S37, the second server 5 determines that the attention area S (see FIG. 15) set in the video 320a is crowded (an example of a predetermined event) based on a determination condition associated with the attention area S. Thereafter, the second server 5 transmits a highlighting instruction including information about the camera 3a-3 capturing the video 320a (e.g., the ID of the camera 3a-3) to the first server 4. Here, the attention area S may be set in the vicinity of the remote customer service device 2a and outside the field of view of the imaging unit 27 of the remote customer service device 2a.

[0108] Furthermore, the area of ​​interest S in the video 320a may be set in response to an input operation by the operator U on the information terminal 6. For example, when the operator U specifies the area of ​​interest S in the video 320a while the video 320a is displayed on the information terminal 6, information indicating the area of ​​interest S in the video 320a (for example, coordinate information defining the area of ​​interest S in the video 320a) is transmitted from the information terminal 6 to the second server 5. Thereafter, the second server 5 sets the area of ​​interest S in the video 320a. Note that the area of ​​interest S may be set automatically or manually to an area within a predetermined distance from the remote customer service device 2a or an area where people have been accumulating or the like with a predetermined frequency or more in the past.

[0109] Furthermore, the determination condition associated with the attention area S is a determination condition related to at least one of the number of people present in the attention area S and their staying time. For example, the determination condition may be a determination condition indicating that the number of people present in the attention area S is N or more (N is a natural number) and that the staying time of the people in the attention area S is M seconds or more (M is a natural number). In this case, the second server 5 may determine that the attention area S is crowded when detecting that the number of people present in the attention area S set in the video 320a is N or more and that the staying time is M seconds or more. Here, the second server 5 may use an image recognition model when detecting a person K present in the attention area S. Note that in this example, congestion in the attention area S is given as an example of a predetermined event in the attention area S, but the predetermined event is not limited to this.

[0110] Returning to FIG. 14, in step S38, the first server 4 identifies the remote customer service device 2a associated with the camera 3a-3 based on the highlighting instruction from the second server 5 and the store management table (see FIG. 9(c)). Next, the first server 4 transmits to the information terminal 6e a highlighting instruction to highlight the thumbnail of the video 110a captured by the remote customer service device 2a (step S39). Thereafter, in step S40, the information terminal 6e highlights the thumbnail of the video 110a captured by the remote customer service device 2a in response to the highlighting instruction transmitted from the first server 4 (see FIG. 15). For example, the highlighting of the thumbnail of the video 110a may be visually achieved by making the outer frame of the thumbnail of the video 110a bold and in a conspicuous color. Note that steps S40 and after are similar to steps S15 to S25 in FIG. 8, and therefore description thereof will be omitted.

[0111] 15, with the thumbnail of image 110a highlighted, image 320a may be superimposed on image 110a as a wipe image. In this case, in step S37, second server 5 may transmit image 320a to first server 4 together with a highlighting instruction including information about camera 3a-3. Note that the wipe image may not only be superimposed on image 110a, but may also be displayed by push notification or pop-up notification. Furthermore, the wipe image may be displayed under the conditions for being highlighted in the example shown in FIG.

[0112] According to this modification, in response to detection of congestion in the attention area S in the video 320a, the remote customer service device 2a associated with the camera 3a-3 capturing the video 320a is identified, and a thumbnail of the video 110a captured by the remote customer service device 2a is highlighted. In this way, the operator Ue can clearly recognize that congestion is occurring in the attention area S of the store A where the remote customer service device 2a is located through the highlighted thumbnail of the video 110a. Therefore, even when there is no one within the field of view of the remote customer service device 2a, the operator Ue can understand the need for remote customer service in the store A, thereby effectively preventing lost sales opportunities due to remote customer service. In this way, a remote customer service system 1 capable of improving the quality of remote customer service can be provided.

[0113] Furthermore, in this modification, the image 320a is superimposed on the image 110a, and therefore the operator Ue can recognize that congestion is occurring in the attention area S of the store A by visually checking the images 320a and 110a simultaneously displayed on the information terminal 6e. In this way, the operator Ue can immediately grasp the need for remote customer service in the store A.

[0114] In this modification, only the fact that the standby screen 100 is displayed on the display unit 66 of the information terminal 6e has been described. However, if the operators Uf and Ug are also logged in to the remote customer service application, the standby screen 100 is also displayed on the display units 66 of the information terminals 6e and 6f. That is, in step S35, the first server 4 transmits data for displaying the standby screen 100 to each of the information terminals 6e to 6g. Similarly, in step S39, a highlighting instruction for highlighting the thumbnail of the video 110a is transmitted to each of the information terminals 6e to 6g. Thereafter, in step S40, each of the information terminals 6e to 6g highlights the thumbnail of the video 110a in response to the highlighting instruction.

[0115] Furthermore, in this modification, the attention area S is set only in the image 320a. However, the attention area S may also be set in other camera images (specifically, images 320b to 320f) other than the image 320a. That is, multiple attention areas S may be set within the store A. Furthermore, while the image 110a displayed on the standby screen 100 is highlighted, the thumbnail of the image 320a displayed on the customer service screen 200 shown in FIG. 12 may also be highlighted. In this case, in step S39 of FIG. 14, the first server 4 transmits a highlighting instruction to the information terminal 6e to highlight the thumbnail of the image 320a. Thereafter, in step S40, the information terminal 6e highlights the thumbnail of the image 320a on the customer service screen 200 in response to the highlighting instruction transmitted from the first server 4. Similarly, when the thumbnail of the image 320a displayed on the customer service screen 200 is highlighted, the operator Ue can clearly recognize that congestion is occurring in the attention area S.

[0116] (Modification in which the video captured by the remote customer service device 2 and the video captured by the camera 3 are displayed interchangeably) In the above description, the video captured by the remote customer service device 2 is displayed as the main image, and the video captured by the camera 3 is displayed as the sub image. However, this main / sub image relationship may be reversed. That is, the video captured by the camera 3 may be displayed as the main image, while the video captured by the remote customer service device 2 may be displayed as the sub image. This is based on the premise that a system in which the operator U views the video captured by the camera 3 on the information terminal 6 via a viewer or the like is used, and the video captured by a certain camera 3 is displayed together with the corresponding video captured by the remote customer service device 2. In this case, in the description from FIG. 6 onwards, the remote customer service device 2 and the camera 3, and the first server 4 and the second server 5 are swapped, respectively. Also, in the data structure of FIG. 9(c), the remote customer service device ID and the camera ID are swapped. In the UI diagrams of FIGS. 10 to 13 and 15, the video captured by the remote customer service device 2 and the video captured by the camera 3 are swapped.

[0117] Although the embodiments of the present invention have been described above, the technical scope of the present invention should not be construed as being limited by the description of the present embodiments. The present embodiments are merely examples, and it will be understood by those skilled in the art that various modifications of the embodiments are possible within the scope of the invention described in the claims. The technical scope of the present invention should be determined based on the scope of the invention described in the claims and its equivalents. [Explanation of symbols]

[0118] 1: remote customer service system, 2, 2a, 2b, 2c: remote customer service device, 3, 3a-1 to 3c-10: camera, 4: first server, 5: second server, 6, 6e, 6f, 6g: information terminal, 7: communication network, 20: communication bus, 21: control unit, 22: storage device, 23: input / output interface, 24: communication unit, 25: input operation unit, 26: display unit, 27: imaging unit, 28: microphone, 29: speaker, 30: communication bus, 31: control unit, 32: storage device, 33: position information acquisition unit, 34: communication unit, 35: input operation unit, 36: imaging unit, 37: PTZ mechanism, 40: communication bus, 41: control unit, 42: storage device, 43: input / output interface, 44: communication unit, 60: communication bus, 61: control unit, 62: storage device, 63: input / output interface, 64: communication unit, 65: input operation unit, 66: display unit, 67: imaging unit, 68: microphone, 69: speaker, 100: standby screen, 100a, 110a, 110b, 11c: video, 120: video display area, 130: operator display area, 200: customer service screen, 202: housing, 210: video display area, 212: store manual display area, 220: operator video display area, 223e: video, 230: store explanation material list area ,240: Chat display area, 250: Camera button, 260: Home button, 263: End customer service button, 265: Hold button, 310: Camera image display area, 320, 320a, 320b, 320c, 320d, 320e, 320f: Image, 340: Slider, 350: Timeline, K: Person, S: Attention area, U, Ue, Uf, Ug: Operator, W1, W2: Rectangular mark

Claims

1. a remote customer service device configured to support remote customer service and to capture a first video showing a customer; a plurality of cameras configured to capture a second image showing a scene in a predetermined area; an information terminal operated by an operator who provides remote customer service; A remote customer service system that provides remote customer service by Identifying at least one camera associated with the remote customer service device from among the plurality of cameras; displaying, on the information terminal, a first image captured by the remote customer service device and a second image captured by the specified camera; Remote customer service system.

2. When a plurality of cameras are identified as cameras associated with the remote customer service device, displaying, on the information terminal, thumbnails of a plurality of second videos captured by the identified plurality of cameras together with the first video; The remote customer service system according to claim 1 .

3. When one of the thumbnails of the plurality of second videos is designated in response to an input operation on the information terminal, displaying, on the information terminal, the second image corresponding to the designated thumbnail in an enlarged state together with the first image; The remote customer service system according to claim 2 .

4. a plurality of remote customer service devices; displaying, on the information terminal, a plurality of first images captured by the plurality of remote customer service devices; When a first remote customer service device is designated from among the plurality of remote customer service devices in response to an input operation on the information terminal, Identifying at least one camera associated with the first remote customer service device; displaying, on the information terminal, a first image captured by the first remote customer service device and a second image captured by the specified camera; The remote customer service system according to claim 1 .

5. a first server capable of communicating with the remote customer service device and the information terminal; a second server capable of communicating with the plurality of cameras and the first server; connected to the first video is provided from the remote customer service device to the information terminal via the first server; the second video is provided from the camera to the information terminal via the second server and the first server; The remote customer service system according to claim 1 .

6. identifying the at least one camera associated with the remote customer service device based on the camera identification information; The remote customer service system according to claim 1 .

7. registering information indicating a correspondence relationship between the camera and the remote customer service device in response to a user instruction; identifying the at least one camera associated with the remote customer service device based on the information indicating the correspondence relationship; The remote customer service system according to claim 1 .

8. analyzing the second video to identify the at least one camera associated with the remote customer service device; The remote customer service system according to claim 1 .

9. highlighting the remote customer service device shown in the second image; The remote customer service system according to claim 1 .

10. Identifying an area where the remote customer service device is located in response to a user instruction; highlighting the area where the remote customer service device is located; The remote customer service system according to claim 9.

11. Identifying an area where the remote customer service device is located by analyzing the second video; highlighting the area where the remote customer service device is located; The remote customer service system according to claim 9.

12. highlighting people present around the remote customer service device shown in the second image; The remote customer service system according to claim 1 .

13. Identifying a surrounding area of ​​the remote customer service device in response to a user instruction; highlighting people present in the surrounding area; The remote customer service system according to claim 12.

14. Detecting a person present in a surrounding area of ​​the remote customer service device by analyzing the second video; highlighting people present in the surrounding area; The remote customer service system according to claim 12.

15. executing a predetermined display process in the information terminal in response to an occurrence of a predetermined event in the second video; The remote customer service system according to claim 1 .

16. the predetermined event is that a determination condition related to at least one of the number of people present in the attention area of ​​the second video and the duration of their stay is satisfied. The remote customer service system according to claim 15.

17. the predetermined display process is to highlight the second video in which the predetermined event has occurred. The remote customer service system according to claim 15.

18. the predetermined display process is to highlight a first image corresponding to a second image in which the predetermined event has occurred. The remote customer service system according to claim 15.

19. the predetermined display process is to display the second video in which the predetermined event occurred together with a first video corresponding to the second video in which the predetermined event occurred. The remote customer service system according to claim 15.

20. the information terminal is configured to capture a third video showing the operator; the remote customer service system displays the third video captured by the information terminal on the remote customer service device, The remote customer service system according to claim 1 .

21. a plurality of remote customer service devices configured to support remote customer service and capture a first video showing a customer; a camera configured to capture a second image showing a scene in a predetermined area; an information terminal operated by an operator who provides remote customer service; A remote customer service system that provides remote customer service by identifying at least one remote customer service device associated with the camera from among the plurality of remote customer service devices; displaying, on the information terminal, the second image captured by the camera and the first image captured by the specified remote customer service device; Remote customer service system.

22. A remote customer service method executed by a remote customer service system, The remote customer service system includes: a remote customer service device configured to support remote customer service and to capture a first video showing a customer; a plurality of cameras configured to capture a second image showing a scene in a predetermined area; an information terminal operated by an operator who provides remote customer service; We provide remote customer service through The remote customer service method includes: identifying at least one camera among the plurality of cameras associated with the remote customer service device; a step of displaying, on the information terminal, a first image captured by the remote customer service device and a second image captured by the specified camera; Including, Remote customer service methods.

23. A remote customer service program that causes a remote customer service system to execute the remote customer service method according to claim 22.

Images