Remote reception system, information processing apparatus, and remote reception method

Abstract

To quickly start remote reception for a user.SOLUTION: A remote reception system comprises a server, and cameras that are arranged in a plurality of stores and can be connected to a user terminal. The server calculates an index value for every camera on the basis of the degree of congestion based on the number of visitors included in a photographable area of every camera and the number of user terminals being connected to every camera, and the number of clerks included in the photographable area of every camera, and selects a camera to be connected to a first user terminal on the basis of the calculated index value. Remote communication is executed between the first user terminal and the selected camera.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a remote customer service system, an information processing device, and a remote customer service method. [Background technology]

[0002] JP 2019-79269 A (Patent Document 1) is a background technology in this technical field. This publication states that "the remote customer service program causes a computer to execute a process to accept, from a customer's terminal, input or selection of information specifying a product and information specifying an area where the product is planned to be used. The remote customer service program causes a computer to execute a process to specify a salesperson related to the area who handles the product. The remote customer service program also causes a computer to execute a process to cause communication between the terminal of the selected salesperson and the customer's terminal" (see abstract). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-79269 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 allows customers to receive remote customer service by matching them with store clerks, but if the matched store clerk is busy, the customer cannot receive remote customer service promptly. Therefore, one aspect of the present invention is to promptly start remote customer service for the user.

[0005] Therefore, one aspect of the present invention is to realize remote customer service that reduces the effort required of customers and prevents loss of sales opportunities. [Means for solving the problem]

[0006] In order to solve the above problem, one aspect of the present invention employs the following configuration: A remote customer service system includes a server and cameras arranged in each of a plurality of stores and connectable to a user terminal, the server retaining camera management information indicating the number of store staff and customers within the imageable area of ​​each of the cameras and the user terminals currently connected to each of the cameras, receiving a store login request from a first user terminal, calculating an index value for each of the cameras based on the number of customers within the imageable area and the number of connected user terminals, and the customer congestion level based on the number of store staff within the imageable area, selecting a camera to connect to the first user terminal based on the index value, and remote communication being executed between the first user terminal and the selected camera. [Effects of the Invention]

[0007] According to one aspect of the present invention, remote customer service for a user can be started promptly.

[0008] Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a configuration example of a remote customer service system according to a first embodiment. [Figure 2] FIG. 2 is a block diagram illustrating a configuration example of an XR (Extended Reality / Cross Reality) server according to the first embodiment. [Figure 3] FIG. 2 is a block diagram illustrating an example of the configuration of a store server according to the first embodiment. [Figure 4A] FIG. 2 is an explanatory diagram showing an example of the layout of a store X in which a store system according to the first embodiment is installed. [Figure 4B] FIG. 2 is an explanatory diagram showing an example of the layout of a store Y in which a store system according to the first embodiment is installed. [Figure 4C]FIG. 2 is an explanatory diagram showing an example of the layout of a store Z in which a store system according to the first embodiment is installed. [Figure 5] FIG. 10 is a diagram illustrating an example of a data configuration of a user management table according to the first embodiment. [Figure 6] FIG. 10 is a diagram illustrating an example of a data configuration of a store clerk management table in the first embodiment. [Figure 7] FIG. 2 is a diagram illustrating an example of a data configuration of a store management table according to the first embodiment. [Figure 8A] FIG. 10 is a diagram illustrating an example of a data configuration of a location information management table of a store X in the first embodiment. [Figure 8B] FIG. 10 is a diagram illustrating an example of a data configuration of a location information management table of a store Y in the first embodiment. [Figure 8C] FIG. 10 is a diagram illustrating an example of a data configuration of a location information management table of a store Z in the first embodiment. [Figure 9A] FIG. 10 is a diagram illustrating an example of the data configuration of a camera management table of a store X in the first embodiment. [Figure 9B] FIG. 10 is a diagram illustrating an example of the data configuration of a camera management table of a store Y in the first embodiment. [Figure 9C] FIG. 10 is a diagram illustrating an example of the data configuration of a camera management table of a store Z in the first embodiment. [Figure 10A] FIG. 10 is a diagram illustrating an example of a data configuration of a merchandise type management table of a store X in the first embodiment. [Figure 10B] FIG. 10 is a diagram illustrating an example of a data configuration of a merchandise type management table of a store Y in the first embodiment. [Figure 10C] FIG. 10 is a diagram illustrating an example of a data configuration of a merchandise type management table of a store Z in the first embodiment. [Figure 11A] FIG. 3 is a sequence diagram illustrating an example of the overall processing of the remote customer service system according to the first embodiment. [Figure 11B] FIG. 3 is a sequence diagram illustrating an example of the overall processing of the remote customer service system according to the first embodiment. [Figure 12] FIG. 10 is a diagram showing an example of the screen configuration of a shopping mall display screen in the first embodiment. [Figure 13] 10 is a flowchart illustrating an example of a user management process according to the first embodiment. [Figure 14] 10 is a flowchart illustrating an example of a store selection process according to the first embodiment. [Figure 15] 10 is a flowchart illustrating an example of a customer service management process according to the first embodiment. [Figure 16] 10 is a flowchart illustrating an example of a camera management table update process in the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In this embodiment, the same components are generally designated by the same reference numerals, and repeated explanations will be omitted. It should be noted that this embodiment is merely an example for realizing the present invention, and does not limit the technical scope of the present invention. [Example]

[0011] FIG. 1 is a block diagram showing an example of the configuration of a remote customer service system. In this embodiment, a brand has multiple stores, each staffed by a salesperson. Furthermore, in this embodiment, each store has one or more products. The salesperson staff at each store provide face-to-face customer service to customers who visit the store, and also provide remote customer service to users in remote locations using the remote customer service system.

[0012] The remote customer service system includes, for example, an XR (Extended Reality / Cross Reality) server 1, one or more user terminals 2, and multiple store systems 40. The XR server 1 is connected to the Internet 7 via a router 6, each user terminal 2 is connected to the Internet 7 via a wireless router 8, and each store system 40 is connected to the Internet 7 via a wireless router 4 included in the store system 40, thereby enabling mutual communication between the XR server 1, the user terminals 2, and each store system 40. Note that the communication method between the XR server 1, the user terminals 2, and the store system 40 is not limited to the example shown in FIG. 1.

[0013] The XR server 1 executes an intermediary process for remote customer service between a user having a user terminal 2 and a store clerk in a store where the store system 40 is installed.

[0014] Each store system 40 is located in a store. The store system 40 includes a store server 3, a wireless router 4, and one or more 360-degree cameras 5. In the example of FIG. 1, each 360-degree camera 5 is a network camera connected to the wireless router 4, but it may also be a webcam connected to the store server 3. Furthermore, instead of the 360-degree camera 5, a camera with a field of view of less than 360 degrees may be used.

[0015] Each 360-degree camera 5 is equipped with (or connected to) a microphone and a speaker. The store server 3 transmits the video and audio captured by each 360-degree camera 5 to the user terminal 2, and outputs the audio received from the user terminal 2 from the 360-degree camera 5, thereby realizing remote customer service between the user with the user terminal 2 and a store clerk in the store where the store system 40 is installed.

[0016] The user terminal 2 may be, for example, a PC (Personal Computer), a smartphone, or a tablet terminal on which an application for realizing specified functions in a virtual space or in a logged-in store is installed, or may be a headset including a goggle-type display device, headphones, a microphone, and a gyro sensor (for example, a VR (Virtual Reality) headset).

[0017] The application realizes functions that allow a user to, for example, enter a virtual space such as a shopping mall (described below) or a login store, move around the virtual space or login store, view objects located in the virtual space or login store, and communicate with store staff located in the login store. For example, when another 360-degree camera 5 displayed on the display device of the user terminal 2 is selected in accordance with input from the user of the user terminal 2, a connection request is sent to the store server 3 to change the connected 360-degree camera 5 to the other 360-degree camera 5, thereby enabling movement within the login store. This allows the user to freely and easily roam around the desired location in the login store.

[0018] 2 is a block diagram showing an example configuration of the XR server 1. The XR server 1 is configured by a computer having, for example, a CPU (Central Processing Unit) 101, a memory 102, an auxiliary storage device 103, a communication device 104, an input device 105, and an output device 106.

[0019] The CPU 101 is an example of a processor, and executes programs stored in the memory 102. The memory 102 includes a ROM (Read Only Memory), which is a nonvolatile storage element, and a RAM (Random Access Memory), which is a volatile storage element. The ROM stores unchanging programs (e.g., a BIOS (Basic Input / Output System)). The RAM is a high-speed, volatile storage element such as a DRAM (Dynamic Random Access Memory), and temporarily stores programs executed by the CPU 101 and data used when the programs are executed.

[0020] The auxiliary storage device 103 is a large-capacity, non-volatile storage device such as a magnetic storage device (HDD (Hard Disk Drive)) or a flash memory (SSD (Solid State Drive)), and stores programs to be executed by the CPU 101 and data to be used when the programs are executed. That is, the programs are read from the auxiliary storage device 103, loaded into the memory 102, and executed by the CPU 101.

[0021] The input device 105 is a device such as a keyboard or mouse that receives input from an operator. The output device 106 is a device such as a display device or printer that outputs the results of program execution in a format that can be viewed by the operator.

[0022] The communication device 104 is a network interface device that controls communication with other devices in accordance with a predetermined protocol. The communication device 104 may also include a serial interface such as a USB (Universal Serial Bus).

[0023] Some or all of the programs executed by the CPU 101 may be provided to the XR server 1 via a network from removable media (CD-ROM, flash memory, etc.), which are non-transitory storage media, or from an external computer equipped with a non-transitory storage device, and stored in a non-volatile auxiliary storage device 103, which is also a non-transitory storage medium. For this reason, the XR server 1 should have an interface for reading data from removable media. This also applies to the user terminal 2 and the store server 3.

[0024] The XR server 1 is a computer system configured on one physical computer or on multiple logically or physically configured computers, and may operate in separate threads on the same computer, or may operate on a virtual computer built on multiple physical computer resources. The same applies to the user terminal 2 and the store server 3.

[0025] The CPU 101 includes, for example, a VR space management unit 111, a user management unit 112, and a store selection unit 113, all of which are functional units. The VR space management unit 111 manages the VR space to be displayed on the user terminal 2. The user management unit 112 manages information related to the user of the user terminal 2. The store selection unit 113 selects a 360-degree camera 5 to be connected to the user terminal 2 from the 360-degree cameras 5 installed in the store.

[0026] For example, the CPU 101 functions as a VR space management unit 111 by operating in accordance with a VR space management program loaded into the memory 102, functions as a user management unit 112 by operating in accordance with a user management unit program loaded into the memory 102, and functions as a store selection unit 113 by operating in accordance with a store selection program loaded into the memory 102. The same relationship between programs and function units applies to the functional units included in the CPU 301 of the store server 3, which will be described later.

[0027] Note that some or all of the functions of the functional units in this embodiment may be realized by hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

[0028] The auxiliary storage device 103 stores, for example, a user management table 121, a store clerk management table 122, a store management table 123, and shopping mall VR space data 124. The user management table 121 shows information about the user of the user terminal 2. The store clerk management table 122 shows information about store clerks in stores. The store management table 123 shows information about stores. The shopping mall VR space data 124 includes data for displaying the shopping mall as a VR space.

[0029] In addition, some or all of the information stored in the auxiliary storage device 103 and the auxiliary storage device 303 of the store server 3, which will be described later, may be stored in the memory 102 and the memory 302, which will be described later, respectively, or may be stored in an external database.

[0030] In this embodiment, the information used by the remote customer service system does not depend on the data structure and may be represented in any data structure, such as a table, a list, a database, or a queue.

[0031] 3 is a block diagram showing an example of the configuration of the store server 3. The store server 3 is configured by a computer having, for example, a CPU 301, memory 302, auxiliary storage device 303, communication device 304, input device 305, and output device 306. Explanation of the hardware of the CPU 301, memory 302, auxiliary storage device 303, communication device 304, input device 305, and output device 306 is omitted because it is the same as the explanation of the hardware of the CPU 101, memory 102, auxiliary storage device 103, communication device 104, input device 105, and output device 106.

[0032] The CPU 301 includes, for example, functional units such as a real-time video distribution unit 311, a camera management table update unit 312, a customer service management unit 313, and a location information management table update unit 314. The real-time video distribution unit 311 distributes video and audio captured by the 360-degree camera 5 to the user terminal 2. The camera management table update unit 312 updates a camera management table 322, which will be described later. The customer service management unit 313 manages information related to remote customer service between the user of the user terminal 2 and a store clerk at a store where a 360-degree camera 5 connected to the user terminal 2 is located. The location information management table update unit 314 updates a location information management table 321, which will be described later.

[0033] The auxiliary storage device 303 stores, for example, a location information management table 321, a camera management table 322, and a product type management table 323. The location information management table 321 stores information indicating the positions of the 360-degree cameras 5, store staff, and customers within the store. The camera management table 322 stores information indicating the imageable area of ​​the 360-degree cameras 5, information indicating the number of store staff and customers within the imageable area, and information indicating the user IDs of the user terminals 2 connected to the 360-degree cameras 5. The product type management table 323 stores information indicating the products placed in the store and the 360-degree cameras 5 that can photograph the products.

[0034] 4A, 4B, and 4C are explanatory diagrams showing examples of the layout of a store in which the store system 40 is installed. Fig. 4A shows the layout of store X of brand C, Fig. 4B shows the layout of store Y of brand C, and Fig. 4C shows the layout of store Z of brand C.

[0035] 4A, 4B, and 4C, the position of 360-degree camera 5 is indicated by a white circle, the position of store clerk 202 is indicated by a black circle, and the position of customer 203 (a customer who actually visits the store) is indicated by a diamond. Also, in Figures 4A, 4B, and 4C, the character string written inside the white circle representing 360-degree camera 5, the character string written inside the black circle representing store clerk 202, and the character string written inside the diamond representing customer 203 indicate the camera ID that identifies 360-degree camera 5, the clerk ID that identifies store clerk 202, and the customer ID that identifies customer 203, respectively.

[0036] Shelves 261 to 270 are installed on floor 200 of store X shown in Fig. 4A. Character strings written inside shelves 261 to 270, shelves 281 to 286, and shelves 291 to 296 in Fig. 4A, Fig. 4B, and Fig. 4C indicate the merchandise displayed on the shelves.

[0037] Additionally, six 360-degree cameras 5 are installed on floor 200, and the camera IDs of these cameras are "C1," "C2," "C3," "C4," "C5," and "C6." Areas 211, 212, 213, 214, 215, and 216 on floor 200 are areas that can be photographed by the 360-degree cameras 5 corresponding to the camera IDs "C1," "C2," "C3," "C4," "C5," and "C6," respectively.

[0038] Therefore, when the user terminal 2 is connected to the 360-degree camera 5 of "C1", it can display shelf 261; when connected to the 360-degree camera 5 of "C2", it can display shelves 263 and 264; when connected to the 360-degree camera 5 of "C3", it can display shelves 267 and 268; when connected to the 360-degree camera 5 of "C4", it can display shelf 262; when connected to the 360-degree camera 5 of "C5", it can display shelves 265 and 266; and when connected to the 360-degree camera 5 of "C6", it can display shelves 269 and 270.

[0039] Furthermore, since there are no store clerks 202 in areas 211, 212, and 215, even if the user terminal 2 is connected to the 360-degree camera 5 of "C1," "C2," or "C5," there is a high possibility that the store clerk 202 will not notice the audio received from the user terminal 2 and output from the 360-degree camera 5. If the store clerk 202 does not notice the audio, the user will not be able to receive remote customer service. Furthermore, since there is one store clerk 202 and one customer 203 in each of areas 213, 214, and 216, there is a possibility that each store clerk 202 is serving the customer 203. Therefore, even if the user terminal 2 is connected to the 360-degree camera 5 of "C3," "C4," or "C6," there is a possibility that the user will not be able to receive remote customer service.

[0040] 4B shows floor 230 of store Y, on which shelves 281 to 286 are installed. Four 360-degree cameras 5 are also installed on floor 230, and these cameras have the camera IDs "C7," "C8," "C9," and "C10." Areas 217, 218, 219, and 220 on floor 200 are areas that can be photographed by the 360-degree cameras 5 corresponding to the camera IDs "C7," "C8," "C9," and "C10," respectively.

[0041] Therefore, when the user terminal 2 is connected to the 360-degree camera 5 of "C7", it can display shelf 281; when connected to the 360-degree camera 5 of "C8", it can display shelves 283 and 284; when connected to the 360-degree camera 5 of "C9", it can display shelf 282; and when connected to the 360-degree camera 5 of "C10", it can display shelves 285 and 286.

[0042] Furthermore, since there is no store clerk 202 in areas 217 and 219, even if the user terminal 2 is connected to the 360-degree camera 5 of "C7" or "C9," there is a high possibility that the store clerk 202 will not notice the audio received from the user terminal 2 and output from the 360-degree camera 5. If the store clerk 202 does not notice the audio, the user will not be able to receive remote customer service. Furthermore, since there is one store clerk 202 and one customer 203 in area 220, there is a possibility that the store clerk 202 is serving the customer 203, so even if the user terminal 2 is connected to the 360-degree camera 5 of "C10," the user may not be able to receive remote customer service. Furthermore, there is a store clerk 202 in area 218, but no customer 203, i.e., the store clerk 202 is not serving a customer. Therefore, if the user terminal 2 is connected to the 360-degree camera 5 of "C8," the user will be able to receive remote customer service.

[0043] 4C shows floor 250 of store Z, on which shelves 291 to 296 are installed. Two 360-degree cameras 5 are also installed on floor 250, and their camera IDs are "C11" and "C12." Areas 221 and 222 on floor 200 are areas that can be photographed by the 360-degree cameras 5 corresponding to camera IDs "C11" and "C12," respectively.

[0044] Therefore, when the user terminal 2 is connected to the 360-degree camera 5 of "C11", it can display shelves 291, 292, and 293, and when connected to the 360-degree camera 5 of "C12", it can display shelves 294, 295, and 296.

[0045] Furthermore, since there is no store clerk 202 in area 221, even if the user terminal 2 is connected to the 360-degree camera 5 of "C11," there is a high possibility that the store clerk 202 will not notice the audio received from the user terminal 2 and output from the 360-degree camera 5. If the store clerk 202 does not notice the audio, the user will not be able to receive remote customer service. Furthermore, since there are two store clerks 202 and two customers 203 in area 222, there is a possibility that both store clerks 202 are serving the customers 203, and therefore even if the user terminal 2 is connected to the 360-degree camera 5 of "C12," there is a possibility that the user will not be able to receive remote customer service.

[0046] Furthermore, the remote customer service system of this embodiment selects the 360-degree camera 5 to be connected to by the user terminal 2 based on factors such as the number of store clerks 202 included in the capture area of ​​the 360-degree camera 5. Therefore, a store clerk such as the store clerk 202 with the store clerk ID "S106", who is located close to the 360-degree camera 5 of "C11" but is outside the area 221 that is the capture area of ​​the 360-degree camera 5 of "C11", is not treated as a store clerk who can provide remote customer service using the 360-degree camera 5.

[0047] 5 is a diagram showing an example of the data configuration of the user management table 121. The user management table 121 includes, for example, a user ID column 1211, a login store ID column 1212, an interested product column 1213, and a connected camera ID column 1214.

[0048] The user ID column 1211 holds a user ID that identifies a user. The login store ID column 1212 holds the store ID (information that identifies the store) to which the user is logged in. The interest product column 1213 holds information that indicates products in which the user is interested. The connected camera ID column 1214 holds the camera ID (information that identifies the 360-degree camera) of the 360-degree camera to which the user terminal 2 used by the user is connected.

[0049] 6 is a diagram showing an example of the data configuration of the clerk management table 122. The clerk management table 122 includes, for example, a clerk ID column 1221, a brand ID column 1222, and a store ID column 1223. The clerk ID indicated in the clerk ID column 1221 is the clerk ID of a clerk currently working at the store indicated by the corresponding store ID of the brand indicated by the corresponding brand ID (information identifying the brand).

[0050] 7 is a diagram showing an example of the data configuration of the store management table 123. The store management table 123 includes, for example, a store ID column 1231, a brand column 1232, and a product column 1233. The product indicated in the product column 1233 is a product of the brand indicated by the corresponding brand ID, which is handled at the store indicated by the corresponding store ID.

[0051] Figures 8A, 8B, and 8C are diagrams showing examples of the data configuration of the location information management table 321. Figure 8A shows the location information management table 321 of store X (i.e., the location information management table 321 held by the store server 3 installed in store X), Figure 8B shows the location information management table 321 of store Y (i.e., the location information management table 321 held by the store server 3 installed in store Y), and Figure 8C shows the location information management table 321 of store Z (i.e., the location information management table 321 held by the store server 3 installed in store Z).

[0052] The location information management table 321 shows, for example, a type column 3211, an ID column 3212, an X coordinate column 3213, and a Y coordinate column 3214. The type column 3211 holds information indicating the type of the object whose position is indicated. The types include, for example, "clerk," "camera," and "visitor." The ID column 3212 holds information indicating the ID of the object whose position is indicated. The ID column 3212 of a record whose type is "clerk" stores a clerk ID, the ID column 3212 of a record whose type is "camera" stores a camera ID, and the ID column 3212 of a record whose type is "visitor" stores a customer ID. The X coordinate column 3213 and Y coordinate column 3214 hold the X coordinate and Y coordinate values ​​of the object's position, respectively.

[0053] 9A, 9B, and 9C are diagrams showing examples of the data configuration of the camera management table 322. Fig. 9A shows the camera management table 322 of store X (i.e., the camera management table 322 held by the store server 3 installed in store X), Fig. 9B shows the camera management table 322 of store Y (i.e., the camera management table 322 held by the store server 3 installed in store Y), and Fig. 9C shows the camera management table 322 of store Z (i.e., the camera management table 322 held by the store server 3 installed in store Z).

[0054] The camera management table 322 includes, for example, a camera ID column 3221, a minimum X value of the photographable area column 3222, a minimum Y value of the photographable area column 3223, a maximum X value of the photographable area column 3224, a maximum Y value of the photographable area column 3225, a number of store staff members in the area column 3226, a number of customers in the area column 3227, and a connected user ID column 3228.

[0055] The camera ID field 3221 holds the camera IDs of the cameras installed in the store. The minimum X value of photographable area field 3222, the minimum Y value of photographable area field 3223, the maximum X value of photographable area field 3224, and the maximum Y value of photographable area field 3225 respectively indicate the minimum X coordinate, maximum X coordinate, minimum Y coordinate, and maximum Y coordinate of the area that can be photographed by the 360-degree camera 5 of the corresponding camera ID. Note that although the shape of the photographable area of ​​the 360-degree camera 5 is rectangular in FIGS. 9A to 9C and the above-mentioned FIGS. 4A to 4C, this shape is not limited to a rectangle.

[0056] The number of store staff in area field 3226 holds information indicating the number of store staff within the capture area of ​​the 360-degree camera 5 of the corresponding camera ID. The number of visitors in area field 3227 holds information indicating the number of visitors within the capture area of ​​the 360-degree camera 5 of the corresponding camera ID. The connected user ID field 3228 holds the user ID of the user using the user terminal 2 connected to the 360-degree camera 5 of the corresponding camera ID.

[0057] Note that the camera management table 322 may store the in-area staff density and the in-area visitor density instead of or in addition to the in-area number of staff and the in-area number of visitors. The in-area staff density is the number of staff per unit area within the captureable area of ​​the 360-degree camera 5 of the corresponding camera ID, and the in-area visitor density is the number of visitors per unit area within the captureable area of ​​the 360-degree camera 5 of the corresponding camera ID. In this case, the in-area staff density and the in-area visitor density may be used instead of the in-area number of staff and the number of visitors in the area in the processing described below (for example, in calculating R in step 824 described below). Also, in this embodiment, an example will be described in which only one user terminal 2 can be connected to one 360-degree camera 5 at a time.

[0058] Figures 10A, 10B, and 10C are diagrams showing examples of the data configuration of the product type management table 323. Figure 10A shows the product type management table 323 of store X (i.e., the product type management table 323 held by the store server 3 installed in store X), Figure 10B shows the product type management table 323 of store Y (i.e., the product type management table 323 held by the store server 3 installed in store Y), and Figure 10C shows the product type management table 323 of store Z (i.e., the product type management table 323 held by the store server 3 installed in store Z).

[0059] The product type management table 323 includes, for example, a product column 3231 and a camera ID column 3232. The product column 3231 shows products in the store that can be photographed by the 360-degree camera 5 with the camera ID indicated in the camera ID column 3232.

[0060] 11A and 11B are sequence diagrams showing an example of the overall processing of the remote customer service system. For example, the processing of FIG. 11B is executed following the processing of FIG. 11A. In the example of FIGS. 11A and 11B, two store systems 40 are shown (i.e., an example with two stores is shown), but three or more store systems 40 may be present. Also, all store systems 40 include a 360-degree camera 5, but in the example of FIGS. 11A and 11B, the 360-degree camera included in one of the store systems 40 is omitted due to space limitations.

[0061] 11A and 11B start, it is assumed that the values ​​of the user ID column 1211 and the interested product column 1213 of the user management table 121, the values ​​of each column of the clerk management table 122, and the values ​​of each column of the store management table 123, all of which are held by the XR server 1, are set in advance. It is also assumed that the values ​​of each column of the record in which the value of the type column 3211 of the location information management table 321 is “camera”, the values ​​of the camera ID column 3221, the photographable area X minimum value column 3222, the photographable area Y minimum value column 3223, the photographable area X maximum value column 3224, and the photographable area Y maximum value column 3225 of the camera management table 322, and the values ​​of each column of the product type management table 323, all of which are held by the store server 3, are set in advance.

[0062] Furthermore, the values ​​of each column of the records in which the value of the type column 3211 in the position information management table 321 is "clerk" and the records in which the value of the type column 3211 is "customer" are constantly updated by the position information management table update unit 314. Specifically, for example, a sensor for detecting people flow, such as a LiDAR (Light Detection And Ranging) sensor, is installed in each store, and the position information management table update unit 314 constantly receives and analyzes information from the sensor to identify the positions of the clerk and customers. Note that, for example, using the sensor information, it is possible to determine that people entering and leaving a predetermined area in the store (e.g., a back room) are clerks, and that people not entering and leaving the predetermined area are customers. Note that the method for identifying the positions of the clerk and customers is not limited to this method.

[0063] Each 360-degree camera 5 connects to the store server 3 included in the store system 40 in which it is included (401). The user terminal 2 sends a shopping mall data request to the XR server 1 in accordance with input from the user (402). Upon receiving the shopping mall data request, the user management unit 112 of the XR server 1 executes user management processing (403), and in the user management processing, sends shopping mall VR space data 124 to the user terminal 2 that sent the shopping mall data request (404). Details of the user management processing will be described later with reference to FIG. 13. Step 404 completes the entry of the user of the user terminal 2 into the shopping mall, which is a virtual space, and the VR space management unit 111 controls the display content, etc. on the user terminal 2 in accordance with instructions from the user terminal 2 used by the user entering the shopping mall.

[0064] The user terminal 2 sends a user store login request to the XR server 1 in accordance with input from the user (405). The user store login request includes, for example, the user ID and information indicating the brand desired by the user. Upon receiving the user store login request, the store selection unit 113 of the XR server 1 executes a store selection process (406), and in the store selection process, sends a camera management table request to each store server 3 of the brand indicated in the user store login request (407). Details of the store selection process will be described later using FIG. 14.

[0065] The camera management table update unit 312 of each store server 3 executes camera management table update processing (408). Details of the camera management table update processing will be described later using Fig. 16. The camera management table update unit 312 of each store server 3 transmits the location information management table 321, the product type management table 323, and the camera management table 322 updated in step 408 to the XR server 1 (409).

[0066] In the store selection process, the store selection unit 113 of the XR server 1 determines the 360-degree camera 5 to which the user terminal 2 will connect based on the information in the received location information management table 321, product type management table 323, and camera management table 322, and sends a connection destination notification to the user terminal 2 indicating the camera ID of the 360-degree camera 5 and the store system 40 in which the 360-degree camera 5 is included (410).

[0067] The user terminal 2 sends a connection request to the store server 3 included in the store system 40 indicated in the connection destination notification, requesting connection to the 360-degree camera 5 with the camera ID indicated in the connection destination notification (411). The customer service management unit 313 of the store server 3 that received the connection request executes customer service management processing (412), and in the customer service management processing, sends an OK response to the user terminal 2 permitting connection to the 360-degree camera 5 with the camera ID indicated in the connection request (413). Upon determining that the connection between the user terminal 2 and the 360-degree camera 5 is complete, the customer service management unit 313 sends a connection completion notification to the XR server 1 (414). The connection completion notification includes information indicating, for example, the user ID and camera ID of the user terminal 2 and 360-degree camera 5 with which the connection has been completed, and the store ID of the store where the 360-degree camera 5 is located. Details of the customer service management processing will be described later using FIG. 15.

[0068] If the customer service management unit 313 determines by referring to the camera management table 322 that there is another user terminal 2 connected to the 360-degree camera 5 with the camera ID indicated in the connection request, it may suspend the start of the processing in step 413 and the processing in step 414 (i.e., enter a connection waiting state) until the connection between the 360-degree camera 5 and the other user terminal 2 is terminated. When suspending the start of the processing in step 413 and the processing in step 414, the customer service management unit 313 may send a wait notification to the user terminal 2 that sent the connection request in step 411. The connection waiting state and wait notification also apply to steps 419 and 420, which will be described later.

[0069] Upon receiving the connection completion notification, the user management unit 112 of the XR server 1 executes user management processing (415). The real-time video distribution unit 311 of the store server 3 connected to the 360-degree camera 5 whose connection was completed in step 414 transmits the video and audio captured by the 360-degree camera 5 to the user terminal 2, and outputs the audio received from the user terminal 2 from the 360-degree camera 5 (416). Step 416 enables remote customer service of the user of the user terminal 2 by a store staff member of the store in which the store system 40 is installed.

[0070] The user terminal 2 can send a connection request to change the currently connected 360-degree camera 5 (within the same store) to the store server 3 of that store in accordance with input from the user (417). The customer service management unit 313 of the store server 3 that receives the connection request executes customer service management processing (418). Steps 419 to 422 are similar to steps 413 to 416, respectively.

[0071] In accordance with the input from the user, the user terminal 2 sends a store exit request to the store server 3 included in the store system 40 that includes the 360-degree camera 5 to which the user terminal 2 is connected (423). The customer service management unit 313 of the store server 3 that received the store exit request executes customer service management processing (424), and in the customer service management processing, sends a conversation data notification to the XR server 1 that includes the conversation data between the user and the store clerk that took place in steps 416 and 422, and the user ID of the user (425).

[0072] Upon receiving the conversation data notification, the user management unit 112 of the XR server 1 executes user management processing (426). Steps 427 to 429 are the same as steps 402 to 404, respectively. The user terminal 2 sends a logout request to the XR server 1 in accordance with input from the user (430). Upon receiving the logout request, the user management unit 112 of the XR server 1 executes user management processing (431), and in the user management processing, sends an OK response to the user terminal 2 that sent the logout request (432).

[0073] FIG. 12 is a diagram showing an example of the screen configuration of a shopping mall display screen displayed on the user terminal 2. The shopping mall display screen 20 displays a shopping mall as a VR space. For example, buttons 22a to 22c and 3D models 23a to 23c are displayed on the shopping mall display screen 20. When a button 22a to 22c is selected, a user store login request indicating the corresponding brand is transmitted to the XR server 1. The 3D models 23a to 23c are 3D models of merchandise of the brand corresponding to the button 22a to 22c, respectively.

[0074] 13 is a flowchart showing an example of the user management processing executed in steps 403, 415, 421, and 426. The user management unit 112 of the XR server 1 receives a message (801). The user management unit 112 determines whether the message received in step 801 is a shopping mall data request from the user terminal 2 (802). If the user management unit 112 determines that the message received in step 801 is a shopping mall data request (802: YES), it starts sending the shopping mall VR space data 124 to the user terminal 2 that sent the shopping mall data request (803), updates the user management table 121 (804), and updates the user management processing.

[0075] If the user management unit 112 determines that the message received in step 801 is not a mall data request from the user terminal 2 (802: NO), it determines whether the message is a connection completion notification from the store server 3 (805). If the user management unit 112 determines that the message received in step 801 is a connection completion notification (805: YES), it transitions to step 804.

[0076] If the user management unit 112 determines that the message received in step 801 is not a connection completion notification (805: NO), it determines whether the message is a conversation data notification from the store server 3 (806). If the user management unit 112 determines that the message received in step 801 is a conversation data notification (806: YES), it estimates the product of interest to the user from the conversation data included in the conversation data notification (809) and transitions to step 804.

[0077] In step 809, the user management unit 112 converts the user's voice (voice transmitted from the user terminal 2) included in the conversation data into text and performs natural language processing on the text using a predetermined algorithm to estimate the product of interest. Specifically, the user management unit 112 extracts words indicating the product and words similar to the product from the text, and determines the product of interest to be a product corresponding to a word whose frequency of appearance in the text is high according to a predetermined condition (e.g., a predetermined number of words in descending order of frequency of appearance, or words whose frequency of appearance is equal to or greater than a predetermined value). The user management unit 112 may also calculate an index indicating the user's level of interest in each product (hereinafter also referred to as interest level). A calculation algorithm is established so that the higher the frequency of appearance of each product, the higher the value of the interest level for the product. These processes can eliminate the need for the user to input information indicating the user's interest in the product.

[0078] If the user management unit 112 determines that the message received in step 801 is not a conversation data notification (806: NO), it determines whether the message is a logout request (810). If the user management unit 112 determines that the message is a logout request (S810: YES), it ends the transmission of the shopping mall VR space data 124 to the user terminal 2 that sent the logout request (807), sends an OK response to the user terminal 2 (808), and transitions to step 804. If the user management unit 112 determines that the message received in step 801 is not a logout request (810: NO), it ends the user management process.

[0079] The update contents in the update process of the user management table 121 in step 804 differ depending on the process performed immediately before transitioning to step 804. When transitioning from step 803 to step 804, the user management unit 112 updates the value of the login store ID column 1212 of the user ID of the user terminal 2 that sent the shopping mall data request in the user management table 121 to "- (shop mall)" and updates the value of the connected camera ID column 1214 to "-".

[0080] If the user management unit 112 determines in step 805 that the message is a connection completion notification and transitions to step 804, it updates the values ​​in the login store ID column 1212 and the connected camera ID column 1214 of the user ID indicated in the connection completion notification in the user management table 121 to the store ID and camera ID indicated in the connection completion notification, respectively.

[0081] When the user management unit 112 transitions from step 809 to step 804, it updates the value of the interested product column 1213 of the user ID indicated in the conversation data notification in the user management table 121 to a value indicating the interested product estimated in step 809. In addition, when the user management unit 112 calculates the interest level in step 809, it stores in the user management table 121 information indicating the interest level for each product corresponding to the user ID indicated in the conversation data notification.

[0082] When the user management unit 112 transitions from step 808 to step 804, it updates the value of the login store ID column 1212 of the user ID of the user terminal 2 that sent the logout request in the user management table 121 to "-" (i.e., a value indicating that the user is not logged in to the shopping mall or any store), and updates the value of the connected camera ID column 1214 to "-".

[0083] 14 is a flowchart showing an example of the store selection process executed in step 406. The store selection unit 113 of the XR server 1 receives a user store login request from the user terminal 2 (821). The store selection unit 113 identifies the store ID of the brand desired by the user indicated in the user store login request from the store management table 123, and transmits a camera management table request to each store server 3 included in the store system 40 of the store ID (822).

[0084] The store selection unit 113 determines whether replies (including the location information management table 321, the product type management table 323, and the camera management table 322 updated in the camera management table update process) have been received from all of the store servers 3 that received the camera management table request (823). If the store selection unit 113 determines that a reply has not been received from at least one of the store servers 3 that received the camera management table request (823: NO), it continues to wait for a reply.

[0085] If the store selection unit 113 determines that it has received replies from all store servers 3 that received the camera management table request (823: YES), it identifies the product of interest corresponding to the user ID of the user terminal 2 that sent the user store login request from the user management table 121, identifies the camera ID corresponding to the identified product of interest from the product type management table 323 included in the reply, and calculates an index R for each 360-degree camera 5 with the identified camera ID (824). Note that the product of interest may be specified by input by the user of the user terminal 2.

[0086] Here, the store selection unit 113 calculates R for each 360-degree camera 5 by referring to the camera management table 322, which includes the camera ID of that 360-degree camera 5, as follows: R = (number of customers in the area + number of connected users) / (number of store staff in the area). If the number of store staff in the area is 0, the store selection unit 113 should set R to the constant M (M is an extremely large value that would not normally be possible). The sum of the number of customers in the area and the number of connected users indicates the degree of customer congestion in the area that can be photographed by that camera.

[0087] The store selection unit 113 determines (825) whether there are multiple stores that have installed the 360-degree camera 5 with the smallest R calculated in step 824. If the store selection unit 113 determines that there is only one store that has installed the 360-degree camera 5 with the smallest R (825: NO), it selects the camera ID of the 360-degree camera 5 with the smallest R (827), sends a connection destination notification indicating the selected camera ID and the store system 40 that includes the 360-degree camera 5 to the user terminal 2 (828), and ends the store selection process.

[0088] If the store selection unit 113 determines that there are multiple stores with 360-degree cameras 5 with the smallest R installed (825: NO), it calculates, for each 360-degree camera 5 with the smallest R, the average value A of the distance between the position of each store clerk within the capture area of ​​that 360-degree camera 5 and the position of that 360-degree camera 5, selects the camera ID with the smallest calculated average value A (826), and proceeds to step 828. The store selection unit 113 identifies the positions of each store clerk within the capture area of ​​the 360-degree camera 5 from the position information management table 321 and the camera management table 322, and identifies the position of the 360-degree camera 5 from the position information management table 321.

[0089] In the above example, the smaller R is, the fewer the number of customers (visitors and remote users) who can be served compared to the number of store staff in the area. Therefore, when a user terminal 2 is connected to a 360-degree camera 5 with a small R, there is a high possibility that there will be a store staff member who can quickly begin remote customer service for the user of the user terminal 2, and visitors in the area will also be able to receive sufficient service, and the store will be able to make effective use of available store staff.

[0090] Furthermore, in the above example, the smaller A is, the more store staff are near the 360-degree camera 5. Therefore, when a user terminal 2 is connected to a 360-degree camera 5 with a small A, there is a particularly high possibility that there will be a store staff member available to quickly begin remote customer service for the user of that user terminal 2. In addition, customers in that area will be able to receive sufficient customer service, and the store will be able to make effective use of available store staff.

[0091] Furthermore, in calculating the above-mentioned R and A, not only the distance but also the area that can be photographed by the 360-degree camera 5 is taken into consideration, and the presence or absence of obstacles such as shelves is also taken into consideration, so that a 360-degree camera 5 that is likely to have a store clerk nearby who can quickly begin remote customer service with the user can be selected.

[0092] In addition, if the store selection unit 113 determines in step 824 that the number of store staff in the area for all 360-degree cameras 5 corresponding to the user's products of interest is zero, it may select the camera ID corresponding to the 360-degree camera for which the value obtained by dividing the total number of customers in the store where the 360-degree camera 5 is located by the total number of store staff is smallest, or it may randomly select a camera ID from the camera IDs of the 360-degree cameras 5 corresponding to the user's products of interest, or it may present a list of 360-degree cameras 5 corresponding to the user's products of interest to the user terminal 2 and select a camera ID according to user input.

[0093] Furthermore, in the above example, the store selection unit 113 selects only from 360-degree cameras 5 corresponding to the user's product of interest, but the 360-degree camera 5 may be selected taking into account the interest level described above. Specifically, for example, the store selection unit 113 calculates R for each 360-degree camera 5 of all camera IDs included in the product type management table 323 included in the reply, obtains the interest level for each product corresponding to the user ID of the user terminal 2 that sent the user store login request, and calculates the product (which may be a weighted sum using a predetermined weight) of the R for each 360-degree camera 5 and the interest level of the product corresponding to that 360-degree camera 5 in the product type management table 323, instead of R in the processing of step 824 described above. This makes it possible to select a 360-degree camera 5 that satisfies both the user's interest and ease of receiving remote customer service.

[0094] 15 is a flowchart showing an example of the customer service management process in steps 412, 418, and 424. The customer service management unit 313 of the store server 3 receives a request from the user terminal 2 (841). The customer service management unit 313 determines whether the request received in step 841 is a connection request or a store exit request (842).

[0095] If the customer service management unit 313 determines that the request received in step 841 is a connection request (842: connection request), it connects the 360-degree camera 5 with the camera ID indicated in the connection request to the user terminal 2 that sent the connection request, and instructs the real-time video distribution unit 311 to start video communication (video and audio communication) (843). The customer service management unit 313 starts recording audio (customer service audio) in the video communication between the 360-degree camera 5 with the camera ID connected in step 843 and the user terminal 2 that sent the connection request (844), and the camera management table update unit 312 updates the value of the connected user ID column 3228 corresponding to the camera ID in the camera management table 322 (848), and ends the customer service management process.

[0096] If the customer service management unit 313 determines that the request received in step 841 is a store exit request (842: store exit request), it ends recording of customer service audio for the user terminal 2 that sent the store exit request and the 360-degree camera 5 that includes the user terminal 2 as a connected user ID in the camera management table 322 (845). The customer service management unit 313 instructs the real-time video distribution unit 311 to end video communication between the user terminal 2 and the 360-degree camera 5 (846). The customer service management unit 313 sends a conversation data notification including the recorded customer service audio to the XR server 1 (847) and proceeds to step 848.

[0097] 16 is a flowchart showing an example of the camera management table update process in step 408. The camera management table update unit 312 receives a camera update table request from the XR server 1 (861). The camera management table update unit 312 references the position information management table 321 that is constantly updated by the position information management table update unit 314, and updates the values ​​of the number of store staff in area column 3226, the number of visitors in area column 3227, and the connected user ID column 3228 of the camera management table 322 (862). The camera management table update unit 312 sends the camera management table 322 updated in step 862 to the XR server 1 (863), and ends the camera management table update process.

[0098] The present invention is not limited to the above-described embodiments, but includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0099] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0100] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0101] 1 XR server, 2 user terminal, 3 store server, 5 360-degree camera, 101 CPU, 102 memory, 103 auxiliary storage device, 104 communication device, 112 user management unit, 113 store selection unit, 121 user management table, 122 store clerk management table, 123 store management table, 301 CPU, 302 memory, 303 auxiliary storage device, 304 communication device, 312 camera management table update unit, 313 customer service management unit, 314 location information management table update unit, 321 location information management table, 322 camera management table, 323 product type management table

Claims

1. A remote customer service system, A server; a camera that is arranged in each of a plurality of stores and that is connectable to a user terminal; The server The camera management information is stored to indicate the number of store staff and customers included in the photographable area of ​​each of the cameras, as well as the user terminals currently connected to each of the cameras; receiving a store login request from a first user terminal; Calculating an index value for each camera based on the number of customers included in the photographable area and the degree of customer congestion based on the number of connected user terminals, and the number of store staff included in the photographable area; Selecting a camera to connect to the first user terminal based on the index value; A remote customer service system in which remote communication is performed between the first user terminal and the selected camera.

2. The remote customer service system according to claim 1, A remote customer service system in which the index value is the sum of the number of visitors included in the photographable area and the number of connected user terminals divided by the number of store staff included in the photographable area.

3. The remote customer service system according to claim 1, The server storing location management information indicating the location of each of the cameras and the location of each of the store clerks; Calculating an average value of the distance between the position of each store clerk included in the photographable area and the position of the camera for each of the cameras; A remote customer service system that selects a camera to connect to the first user terminal based on the index value and the average value.

4. The remote customer service system according to claim 1, The server User management information including information indicating an interest in a product by a user of the user terminal; and storing product management information indicating products that can be photographed by each of the cameras; A remote customer service system that selects a camera to connect to the first user terminal based on the information indicating the interests included in the user management information, the products that the camera can photograph, and the index value.

5. The remote customer service system according to claim 4, The user management information indicates an interest product in which the user is interested, The server refers to the user management information and the product management information to identify the camera capable of photographing the product of interest, A remote customer service system that selects a camera to connect to the first user terminal from the identified cameras based on the index value.

6. The remote customer service system according to claim 4, The user management information holds an interest level indicating the strength of interest the user has in each product, A remote customer service system in which the server selects a camera to connect to the first user terminal based on the index value and the degree of interest in merchandise that the camera can photograph.

7. The remote customer service system according to claim 4, The server acquiring conversation data in the remote communication performed between the first user terminal and the selected camera; generating information indicating the interest of the user of the first user terminal in the product based on the frequency of appearance of the product in the conversation data; The remote customer service system stores the generated information indicating interest in the user management information.

8. The remote customer service system according to claim 1, The first user terminal is further provided, The first user terminal an input device; accepting, via an input to the input device, a selection of another camera in the video captured by the camera connected to the first user terminal and displayed on the first user terminal; A remote customer service system that changes the camera connected to the first user terminal to the other camera.

9. An information processing device, a processor and a memory, the memory stores camera management information indicating the number of store staff and the number of customers included in the photographable area of ​​each camera that is arranged in each of a plurality of stores and that can be connected to a user terminal, and indicating the user terminals currently connected to each of the cameras; The processor: receiving a store login request from a first user terminal; Calculating an index value for each camera based on the number of customers included in the photographable area and the degree of customer congestion based on the number of connected user terminals, and the number of store staff included in the photographable area; An information processing device that selects a camera to connect to the first user terminal based on the index value for remote communication between the first user terminal and the selected camera.

10. A remote customer service method using a remote customer service system, The remote customer service system includes: A server; a camera that is arranged in each of a plurality of stores and that is connectable to a user terminal; The server stores camera management information indicating the number of store staff and customers included in the photographable area of ​​each of the cameras, as well as the number of user terminals currently connected to each of the cameras; The remote customer service method includes: the server receives a store login request from a first user terminal; The server calculates an index value for each camera based on the number of customers included in the photographable area and the degree of customer congestion based on the number of connected user terminals, and the number of store staff included in the photographable area; the server selects a camera to connect to the first user terminal based on the index value; A remote customer service method, in which remote communication is performed between the first user terminal and the selected camera.

Images