Virtual reality processing server and virtual reality processing method

Abstract

When a customer visits a virtual facility installed in a virtual space, this virtual reality processing server transmits a video in which a staff avatar is placed in the virtual facility to a customer terminal operated by the customer, controls the movement of the staff avatar in an automatic service mode in which response content for the customer is generated on the basis of dialog log data between the customer and the staff avatar, and determines whether the customer is a specific customer who requires a response from a real staff member. If it is determined that this customer is the specific customer, the virtual reality processing server transmits, to a staff terminal operated by a staff member, notification data notifying the staff member that the customer is the specific customer, and switches to a manual service mode in which the staff member's voice is spoken from the staff avatar.

Description

Virtual reality processing server and virtual reality processing method

[0001] The present invention relates to a virtual reality processing server and a virtual reality processing method.

[0002] 2. Description of the Related Art In recent years, the provision of services built within computers and computer networks has become widespread, and related technological developments have been actively pursued.

[0003] For example, Patent Document 1, entitled "Online game providing system and method using storage medium," describes "duplicating game files stored on a storage medium and each patch file stored in the My Documents directory to the generated directory, and executing a game program executable file from among the duplicated game files" (Claim 1).

[0004] Furthermore, Patent Document 2 states that "in cases where a chatbot that operates on a message service where users exchange messages in a chat room can only exist once on the same message service, it is possible for the user to receive a message response from a chatbot that does not exist on the message service, as desired by the user" (abstract).

[0005] JP 2011-526825 A JP 2020-155000 A

[0006] For example, if a system copy such as that described in Patent Document 1 is applied to a virtual store set up in a virtual space, the store space, where avatars called rooms are placed, can be copied and a virtual store room, called a room, can be created for each customer, allowing them to enjoy leisurely shopping in a virtual store without other customers. Furthermore, by utilizing the chatbot described in Patent Document 2, customer service can be provided through automated responses in the virtual store. By applying the technologies described in Patent Documents 1 and 2 to a virtual store, an environment can be created in which a virtual store dedicated to each customer, with no other customers present, and automated responses by a chatbot can be provided. However, even if customers desire timely prompts and advice from professionals or staff with specialized knowledge, automated responses by a chatbot are unable to provide complex responses. On the other hand, if real-life specialized staff are placed in a virtual store to provide interactive responses to customers in a virtual space, the number of specialized staff that can be placed in the virtual store is limited due to management issues, even if the number of rooms increases with the number of customers. Therefore, the development of a customer service system suited to the actual conditions of a virtual store is desired. This situation is not limited to virtual stores, but is a problem that exists in virtual facilities set up in virtual spaces.

[0007] The present invention has been made to solve the above-mentioned problems, and aims to provide a virtual reality processing server and a virtual reality processing method that can achieve both customer satisfaction for customers visiting a virtual facility and efficient management of the virtual facility.

[0008] In order to achieve the above object, the present invention comprises the configurations set forth in the claims. As one example, the present invention is a virtual reality processing server comprising a processor, a communication device, and a memory, wherein the communication device is connected to a customer terminal operated by a customer visiting a virtual facility set up in a virtual space, and a staff terminal operated by real staff, and the memory stores base facility information which is the original of the video information of the virtual facility, and when the customer visits the virtual facility, the processor reads the base facility information and assigns the base facility to the customer, transmits a video of a staff avatar placed at the base facility to the customer terminal, controls the operation of the staff avatar in an auto customer service mode which generates a response to the customer based on conversation log data between the customer and the staff avatar, determines based on the conversation log data whether the customer is a specific customer who requires an actual response from the staff, and if it is determined that the customer is the specific customer, sends notification data to the staff terminal informing the staff that the customer is the specific customer, and switches the operation control of the staff avatar to a manual customer service mode in which the staff avatar speaks the voice of the staff received from the staff terminal.

[0009] According to the present invention, a virtual reality processing server and a virtual reality processing method can be provided that can achieve both customer satisfaction for customers visiting a virtual facility and efficient management of the virtual facility. In addition to the above, the objectives, configurations, and effects will be clarified in the following embodiments.

[0010] 1 is a schematic diagram of a virtual store customer service system. A configuration diagram showing an example of a glasses-type (transmissive) HMD. A configuration diagram showing an example of an immersive (non-transmissive) HMD. A hardware configuration diagram of an HMD. A hardware configuration diagram of a virtual store server. A functional block diagram of the virtual store server. A flowchart showing the processing flow of the virtual store customer service system. A flowchart showing store duplication processing when a customer visits a store. A diagram showing a state where there are no customers in the base store. A diagram showing a state where there are customers in the base store. A diagram showing a state where there is an additional customer in the first duplicate store. A diagram showing a state where there is an additional customer in the base store. A diagram showing a state where there is an additional customer in the second duplicate store. A diagram showing a state where a real store clerk controls a store clerk avatar. A flowchart showing the flow of a specific customer judgment processing. A diagram showing an example of a monitoring screen of a store clerk terminal. A schematic diagram of a judgment criteria table used in the specific customer judgment processing. A flowchart showing the details of the specific customer judgment processing. A schematic diagram showing an example of a screen display displayed on a customer terminal.

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In each drawing for explaining an embodiment, the same components are generally designated by the same reference numerals, and repeated description thereof will be omitted.

[0012] The present invention is expected to contribute to smooth communication between customers and staff visiting facilities set up in virtual spaces. Therefore, the present invention is expected to improve technology for labor-intensive industries that increase the comfort of virtual spaces, and is therefore expected to contribute to goal 8.2 of the United Nations' Sustainable Development Goals (SDGs) (increasing economic productivity through diversification, technological improvement, and innovation, particularly in industries that increase the value of goods and services and labor-intensive industries).

[0013] The following describes an embodiment in which the present invention is applied to a virtual store customer service system. In the following embodiment, the virtual store corresponds to a virtual facility. The actual store clerks who operate the virtual store correspond to real staff, and the avatars of the clerks correspond to virtual staff. The virtual store server corresponds to a virtual reality processing server.

[0014] The virtual facility is not limited to a virtual store, and may be any type of facility such as a museum, art gallery, aquarium, zoo, historical site, school, or cram school set up in a virtual reality space. The present invention can be applied to a virtual space system in which facility staff, such as guides, commentators, and teachers, are assigned to the virtual facility and the facility staff serve multiple customers.

[0015] First Embodiment FIG. 1 is a schematic diagram of a virtual store customer service system.

[0016] 1, one sales clerk CL serves a first customer CS1 and a second customer CS2 in the virtual store customer service system 100. The number of sales clerks and the number of customers are not limited to this.

[0017] In the virtual store customer service system 100 of Figure 1, HMDG1 worn by store clerk CL is connected to communication network 13 via wireless router R1. HMDG2 worn by first customer CS1 is connected to communication network 13 via wireless router R2. HMDG3 worn by second customer CS2 is connected to communication network 13 via wireless router R3. Furthermore, a virtual store server 14 is connected to communication network 13. HMDG1 corresponds to a staff terminal, and HMDG2 and HMDG3 correspond to customer terminals.

[0018] The store clerk CL, the first customer CS1, and the second customer CS2 log in to the virtual store server 14 from HMDG1, HMDG2, and HMDG3, respectively, to do online shopping and provide customer service.

[0019] The virtual store server 14 distributes various types of video information, such as video of the virtual store including the exterior and interior of the virtual store, objects representing objects within the virtual store, avatar videos of store clerks placed in the virtual store, avatar videos of the first customer CS1, and avatar videos of the second customer CS2, and live data such as the speech of the store clerk CL and the speech of the first and second customers, to HMDG1, HMDG2, and HMDG3. HMDG1, HMDG2, and HMDG3 display the video on the display screen of each HMD and output the audio from the speakers of each HMD.

[0020] Furthermore, the virtual store server 14 manages multiple pieces of information acquired via the communication network 13. The information managed by the virtual store server 14 is, for example, information about users, which will be described later. The information about users includes movement information about HMDG1 and voice information about the store clerk CL, movement information about HMDG2 (movement information about the first customer CS1) and voice information about the first customer CS1, and movement information about HMDG3 (movement information about the second customer CS2) and voice information about the second customer CS2.

[0021] The user's movement information is shown as vector information based on sensor outputs detected by sensors mounted on the HMD worn by the user in response to each user's movement, such as shaking the head, standing, sitting, or moving.

[0022] The customer avatar corresponding to the first customer CS1 and the customer avatar corresponding to the second customer CS2 are displayed with movements corresponding to the vector information corresponding to the respective movements of the first customer CS1 and the second customer CS2.

[0023] The movement control modes of the store clerk avatar include an automatic customer service mode in which customer service is performed using messages automatically generated by the virtual store server 14, and a manual customer service mode in which customer service is performed using messages using movement information and voice information of the store clerk CL. In the manual customer service mode, movement is controlled by the movement information of the HMDG1, but in the automatic customer service mode, movement is controlled by the virtual store server 14 regardless of the direction and timing of the HMDG1 movement. Details of each movement control mode will be described later.

[0024] (Automatic Customer Service Mode) In the automatic customer service mode, in which the store clerk avatar sends an automatic response message back to the customer avatar, the store clerk avatar functions as a virtual store clerk located in the virtual store. In the automatic customer service mode, the virtual store server 14 performs automatic operation control of the store clerk avatar, without following the operation information of the HMDG 1. In the automatic operation control, for example, the virtual store server 14 moves the position of the store clerk avatar so that it faces the customer avatar, and positions the store clerk avatar so that its front face faces the customer avatar. Therefore, even if the operation information of the HMDG 1 does not face the customer avatar, the store clerk avatar is placed in the virtual store room so that it faces the customer avatar.

[0025] (Manual Customer Service Mode) On the other hand, in the manual customer service mode, the store clerk CL serves customers by speaking messages input from the HMDG 1. The movement of the store clerk avatar is controlled in such a way that the movement information of the HMDG 1 is reflected.

[0026] Furthermore, the virtual store server 14 stores information about users, such as user identification information including a nickname or handle name, and avatar image information.

[0027] With this system configuration, each of the store clerk CL, the first customer CS1, and the second customer CS2 can serve customers or shop online while viewing an image of a virtual store in which the avatar of another person different from themselves is superimposed.

[0028] FIG. 2 is a diagram showing the configuration of an example of an eyeglass-type (transmission type) HMD.

[0029] The HMDG1 shown in FIG. 2 is housed in a glasses-like housing G10 and includes a left display 72L and a right display 72R, each of which includes a display surface. The left display 72L and the right display 72R are, for example, transmissive displays. A real image of the outside world is transmitted through the display surfaces of the left display 72L and the right display 72R, and a computer-generated image is superimposed on the real image. The housing G10 is equipped with a control device 11, a camera 71, a communication device 6, a sensor device 5 including various other sensors, and the like. The control device 11 includes a processor 2, a bus 3, and a memory 4, which will be described later. The control device 11 may further include a voice recognition unit 82, a decoder 83, and an encoder 84. The HMDGs 2 and 3 have the same configuration as the HMDG 1, and therefore will not be described here.

[0030] FIG. 3 is a diagram showing the configuration of an example of an immersive (non-transmissive) HMD.

[0031] The immersive HMDG 1a shown in Figure 3 differs significantly from the eyeglass-type HMDG 1 in that the right display 72R and left display 72L are non-transparent. Therefore, the HMDG 1a has a through mode as a control mode. A user wearing the HMDG 1a cannot directly view the outside world. When viewing the outside world while wearing the HMDG 1a, the user switches to the through mode, which displays, for example, an image captured by the camera 71 on the right display 72R and left display 72L.

[0032] In the case of a transparent HMDG1, the operating state in which computer-generated images are not superimposed, or in which the displayed image is displayed in the corner of the field of view, for example, and the real image outside can be easily seen, is referred to as the through mode.

[0033] 3, a processor, a communication device, a sensor device 5 including various other sensors, and the like are not shown, but are assumed to be installed in the same manner as in the case of FIG.

[0034] The HMDG 1a may be connected for communication with a separately configured information device, such as a smartphone 1, and the information device may be used as a controller for the HMDG 1a.

[0035] The HMD used in the virtual store customer service system 100 may be the see-through HMD shown in FIG. 2 or the non-see-through HMD shown in FIG.

[0036] Fig. 4 is a diagram showing the hardware configuration of an HMD. Although Fig. 3 shows the HMD G1 as an example, a non-transparent HMD also has a similar configuration.

[0037] As shown in FIG. 4, the HMDG 1 includes a processor 2, a bus 3, a memory 4, a sensor device 5, a communication device 6, a video processing device 7, an audio processing device 8, an operation input device 9, and a gaze detection device 10.

[0038] The processor 2 is a microprocessor unit that, in accordance with a predetermined operating program, controls the entire HMDG 1. The processor 2 mainly processes input from the store clerk CL, who is the user of the HMDG 1, sends and receives information to and from the virtual store server 14 in response to that input, controls the system for processing the virtual store customer service system 100 based on that information, generates images to be displayed, and controls the display.

[0039] The bus 3 is a data communication path for transmitting and receiving various commands and data between the processor 2 and each component block within the HMDG 1 .

[0040] The memory 4 is composed of a program storage area 41 that stores programs for controlling the operation of the HMDG 1, a data storage area 42 that stores various data including operation setting values, detection values ​​from the sensor device 5 described below, images for display, characters, etc., and a rewritable work area 43 that stores work areas used in various program operations.

[0041] The memory 4 includes a volatile memory and a non-volatile memory.

[0042] The RAM is provided as a volatile memory, and a work area 43 is formed in the RAM.

[0043] The nonvolatile memory includes, for example, a readable / writable nonvolatile storage medium such as a semiconductor device memory such as a flash memory or an SSD (Solid State Drive), and a ROM. A magnetic disk drive such as an HDD (Hard Disc Drive) may also be provided as a nonvolatile storage medium. This allows the HMDG1 to retain stored information even when power is not being supplied from an external source.

[0044] The nonvolatile storage medium can store operation programs downloaded from the communication network 13, various data created by executing the operation programs, downloaded content such as video, still images, and audio, and data such as video and still images captured using the camera 71. Each operation program stored in the nonvolatile storage medium can be updated and its functions expanded by downloading from a program server (not shown).

[0045] The sensor device 5 is a general term for various sensors used to detect the state of the HMDG 1. The sensor device 5 includes a GPS (Global Positioning System) sensor 51, a geomagnetic sensor 52, a gyro sensor 53, an acceleration sensor 54, and an attachment / detachment detection sensor 55.

[0046] Based on the sensor outputs from the GPS sensor 51, the geomagnetic sensor 52, the gyro sensor 53, and the acceleration sensor 54, the position, tilt, direction, and movement of the HMDG 1 are measured.

[0047] Furthermore, the store clerk CL can detect whether the HMDG 1 is attached or detached based on the sensor output of the attachment / detachment detection sensor 55. Depending on the attachment / detachment state, a notification of the detection of a specific customer from the virtual store server 14 may be transferred from the HMDG 1 to a smartphone or the like.

[0048] The attachment / detachment detection sensor 55 may be any type of sensor that can detect that the HMDG1 is attached to the head of the store clerk CL, such as a pressure sensor, touch sensor, or photo sensor located inside the HMD.

[0049] The communication device 6 includes a LAN (Local Area Network) communication device 61 , a mobile wireless communication device 62 , and a close-proximity wireless communication device 63 .

[0050] The LAN communication device 61 connects to external devices via a communication network 13 such as the Internet via an access point, a wireless router, etc. Therefore, the LAN communication device 61 corresponds to a first communication device. The LAN communication device 61 may be a wireless connection device such as Wi-Fi (registered trademark). As a result, the HMDG 1 wirelessly connects to an access point, a wireless router, etc. via the LAN communication device 61.

[0051] The mobile wireless communication device 62 performs telephone communication (calls) and transmits and receives data via the communication network 13 by wireless communication with a base station of a mobile wireless communication network (not shown). Communication with a base station or the like may be performed by other communication methods such as the W-CDMA (Wideband Code Division Multiple Access) (registered trademark) system, the GSM (Global System for Mobile communications) (registered trademark) system, the LTE (Long Term Evolution) system, or 4G, 5G, etc. The mobile wireless communication device 62 can also communicate with external devices and therefore corresponds to a first communication device.

[0052] The LAN communication device 61 and the mobile radio communication device 62 each include an encoding circuit, a decoding circuit, an antenna, and the like.

[0053] The close proximity wireless communication device 63 has a communication function of the Bluetooth (registered trademark) system, for example, but is not particularly limited to this, and other communication systems such as infrared communication may also be used. The close proximity wireless communication device 63 corresponds to a second communication device for connecting wirelessly to the mobile information terminal that is the notification target.

[0054] The image processing device 7 is composed of a camera 71, a right display 72R, and a left display 72L. The camera 71 is a camera unit that inputs visible image data of the surroundings and objects by converting visible light input from a lens into an electrical signal using an electronic device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0055] The camera 71 may further include a TOF (Time of Flight) sensor capable of capturing a distance to an object being imaged as a distance image. This allows for accurate detection using the visible image and distance image, such as selecting an object by pointing at multiple images displayed on the HMDG 1 and holding the hand over the image and pinching it with the thumb and index finger (hereinafter referred to as "pointing"). Using these visible and distance images, the distance to an object the user is gazing at can also be measured. The surroundings can also be scanned, and a three-dimensional space map can be generated using the output of the sensor device 5 and the image processing device 7.

[0056] The right display 72R and the left display 72L each display an image by irradiating the display surface of the right display 72R and the left display 72L with projection light obtained from a display device such as a liquid crystal panel. The right display 72R and the left display 72L each include a video RAM (not shown). Images are displayed on the display screen based on the image data input to the video RAM.

[0057] The voice processing device 8 includes a microphone 81, a voice recognition unit 82, a decoder 83, an encoder 84, a right speaker 85R, and a left speaker 85L.

[0058] The microphone 81 converts the user's voice or the like into audio data and inputs it.

[0059] The right speaker 85R and the left speaker 85L output audio information and the like required by the user.

[0060] The voice recognition unit 82 analyzes the input voice information and extracts instruction commands, etc. The voice recognition unit 82 may be used to operate the HMDG 1 using voice commands for operation instructions, or may be used to analyze the content of the voice conversations of each user in a conference.

[0061] The decoder 83 has functions such as decoding the encoded audio signal (audio synthesis processing) and a three-dimensional audio processing function according to the transmission characteristics of the audio, and outputs three-dimensional audio to the user of the HMDG 1 from the right speaker 85R and the left speaker 85L.

[0062] The encoder 84 performs encoding processing on the input audio information to generate an encoded audio signal.

[0063] The operation input device 9 is a user interface for inputting operation instructions to the HMDG 1. The operation input device 9 may be an operation key with an array of button switches or the like, or may be configured as a device for inputting and analyzing gesture movements. Furthermore, the operation input device 9 may be configured as a separate mobile terminal device connected via the communication device 6 by wired or wireless communication.

[0064] The gaze detection device 10 detects the gaze direction of a user wearing the HMDG 1. The right gaze detection sensor 10R and the left gaze detection sensor 10L may detect the gaze by irradiating the user's eyes with invisible light (such as infrared light) and then processing the pupil image obtained from the captured image, but the detection method is not particularly limited.

[0065] FIG. 5 is a diagram showing the hardware configuration of the virtual store server 14. As shown in FIG.

[0066] The virtual store server 14 includes a processor 201 , a RAM 202 , a ROM 203 , a HDD 204 , an input I / F 205 , an output I / F 206 , and a communication I / F 207 , which are connected to each other via a bus 208 .

[0067] An input device 211 such as a keyboard is connected to the input I / F 205 .

[0068] The output I / F 206 is connected to an output device 212 such as a display.

[0069] The communication I / F 207 is connected to a store clerk terminal HMDG1 and customer terminals HMDG2 and HMDG3.

[0070] The processor 201 may be a CPU, or a GPU if an AI engine is used to control the chatbot.

[0071] FIG. 6 is a functional block diagram of the virtual store server 14.

[0072] The virtual store server 14 includes a login processing unit 301, a virtual store setting unit 302, a chatbot control unit 303, a customer avatar control unit 304, a synthetic voice generation unit 305, a base store information storage unit 306, a judgment criteria table storage unit 307, a synthetic voice information storage unit 308, and a communication control unit 309.

[0073] The login processing unit 301, virtual store setting unit 302, chatbot control unit 303, customer avatar control unit 304, synthetic voice generation unit 305, and communication control unit 309 are configured by hardware and software working together when the processor 201 loads a virtual store customer service program stored in the ROM 203 or HDD 204 into the RAM 202 and executes it. The function of each unit will be described later.

[0074] The base store information storage unit 306, the determination criteria table storage unit 307, and the synthetic voice information storage unit 308 are configured using partial areas of a non-volatile memory such as the HDD 204.

[0075] The flow of processing in the virtual store customer service system will be described with reference to Fig. 7. Fig. 7 is a flowchart showing the flow of processing in the virtual store customer service system.

[0076] As shown in FIG. 7, when the virtual store opens, a store clerk CL logs in to the virtual store server 14, and the login processing unit 301 performs login authentication of the store clerk CL (S01).

[0077] The base store information storage unit 306 stores in advance base store information that includes the appearance of the virtual store, the store space, and the placement of product objects. The virtual store setting unit 302 reads out the base store information and generates a base store (S02).

[0078] The chatbot control unit 303 places the store clerk avatar A_CL in the base store and waits there (S03). The chatbot control unit 303 controls the display position of the store clerk avatar A_CL, controls switching between the automatic customer service mode and the manual customer service mode, and executes the specific customer judgment process required for the switching control. Details will be described later.

[0079] The customer logs in to the virtual store server 14, and the login processing unit 301 performs login authentication of the customer (S04).

[0080] The virtual store setting unit 302 assigns an available virtual store to the customer (S05). The virtual store setting unit 302 first assigns a base store to the customer, but if the base store is not available, it copies the base store to create a duplicate store. The process of creating a duplicate store by the virtual store setting unit 302 and allocating the virtual store to the customer will be described in detail later.

[0081] The customer avatar control unit 304 displays the customer avatar in the assigned virtual store (S06). From now on, the customer avatar control unit 304 also controls the movement of the customer avatar.

[0082] If the first customer after the store opens is customer 1 CS1, then customer 1 CS1 is the only customer allowed to enter the base store. As an extension process, if there are other customers categorized as the same group as customer 1 CS1, then customers in the same group as customer 1 CS1 can enter the base store at the same time. In this case, too, only one group can enter the base store at the same time.

[0083] The chatbot control unit 303 starts serving customers in the automatic customer service mode and starts the specific customer determination process (S07). The specific customer determination process will be described in detail later. In the automatic customer service mode, the store clerk avatar responds to messages from the customer avatar using the chatbot. For example, it utters set phrases such as "Welcome," "Please take your time looking around!", and "We close at 8 p.m."

[0084] The chatbot control unit 303 controls the movement of the store clerk avatar A_CL to an appropriate position depending on the movements of the customer avatar.

[0085] When the chatbot control unit 303 detects the specific customer (S08: Yes), it notifies the store clerk CL and switches the operation mode of the store clerk avatar A_CL to manual customer service mode (S09). In manual customer service mode, the virtual store server 14 mediates the conversation between the store clerk terminal HMDG1 and the customer terminal worn by the specific customer, establishing a conversation between the store clerk and the customer.

[0086] Specific customers are defined as customers likely to make a purchase, customers who are judged to have limitations in terms of how they can be served through chat, etc. One of the features of this virtual store customer service system is that it can monitor and identify specific customers, support reporting to store staff (specialized staff), and seamlessly switch the service provided to promising customers from chatbots to actual staff.

[0087] If the specific customer is not detected (S08: No), the customer service continues in the automatic customer service mode.

[0088] Until the customer leaves the store (S10: No), the process returns to step S08 and continues to monitor whether the customer is a specific customer.

[0089] When the customer leaves the store (S10: Yes), the virtual store setting unit 302 sets the virtual store where the customer was located as vacant (S11), and waits for additional customers to enter the store (S04) until the store closes (S12: No). When the virtual store closing process is executed (S12: Yes), the process ends.

[0090] FIG. 8 is a flowchart showing the store duplication process when a customer visits a store.

[0091] The store clerk CL accesses the virtual store server 14 from the HMDG 1, and the login processing unit 301 executes login processing for the store clerk CL, and the virtual store opens.

[0092] The chatbot control unit 303 sets the operation control mode to the automatic customer service mode, and the bot avatar is ready to serve customers (S101). Fig. 9A shows an initial image of a virtual store. In Fig. 9A, only the store clerk avatar A_CL is placed in the base store SH0, and no customers are present.

[0093] The virtual store waits until a customer arrives (S102: No).

[0094] The first customer CS1 accesses the virtual store server 14 from the HMDG2, and when the login processing unit 301 executes the login processing for the first customer CS1 and the first customer CS1 visits the virtual store (S102: Yes), the virtual store setting unit 302 assigns a base store to the first customer CS1 and serves the customer at the base store (S103).

[0095] The base store here refers to a virtual store that is pre-stored in the virtual store server 14. The virtual store server 14 has virtual store information for the base store, including information on the appearance of the virtual store, the interior space, and the layout of product objects sold in the virtual store. By loading the virtual store information into the virtual store server 14, a virtual store is set up in the virtual space. This allows logged-in customers to enter the virtual store via their own customer terminals. The virtual store setting unit 302 waits until additional customers arrive (S104: No).

[0096] In FIG. 9B, a store clerk avatar A_CL and a customer avatar A_CS1 corresponding to a first customer CS1 are located in the base store SH0.

[0097] When an additional customer, for example, second customer CS2, arrives at the virtual store (S104: Yes), if the customer who visited the base store, in the above example, first customer CS1, has already left the store, for example, logged out from the virtual store server 14, and the base store is available (S105: No), the process returns to step S103, and second customer CS2 is served at the base store.

[0098] On the other hand, if the customer is currently serving at the base store, i.e., if the first customer CS1 is currently being served (S105: Yes), the virtual store setting unit 302 copies the base store information to create a duplicate store, and serves the second customer CS2 at this duplicate store (referred to as the first duplicate store) (S106). The duplicate store, like the base store, has a store clerk avatar A_CL placed there. And, like the base store, the store clerk avatar A_CL starts in auto-customer service mode. The virtual store setting unit 302 waits until an additional customer arrives (S107: No).

[0099] In FIG. 9C, a store clerk avatar A_CL and a customer avatar A_CS2 corresponding to a second customer CS2 are placed in a first duplicate store SH1.

[0100] When an additional customer (referred to as a third customer) arrives at the virtual store (S107: Yes), if there is a vacant store in the base store or the duplicate store (S108: Yes), the virtual store setting unit 302 assigns the vacant store, i.e., the base store or the first duplicate store, to the additional customer (third customer). The additional customer (third customer) is then served at the vacant store (S109).

[0101] In FIG. 9D, a store clerk avatar A_CL and a customer avatar A_CS3 corresponding to a third customer CS3 are located in the base store SH0.

[0102] When an additional customer (third customer) comes to the virtual store (S107: Yes), if there is no vacancy in the base store or the duplicate store (S108: No), the process returns to step S106, and another duplicate store, i.e., a second duplicate store, is created. The newly created second duplicate store then serves the additional customer (third customer).

[0103] In FIG. 9E, a store clerk avatar A_CL and a customer avatar A_CS3 corresponding to a third customer CS3 are placed in a second duplicate store SH2.

[0104] FIG. 10 is a diagram showing a state in which a real store clerk controls a store clerk avatar.

[0105] The store clerk CL simultaneously monitors multiple virtual stores, including the base store, the first duplicate store, and the second duplicate store, via the store clerk terminal HMDG1. When the store clerk avatar A_CL placed in each virtual store, including the base store, the first duplicate store, and the multiple stores 2, switches to manual customer service mode, the store clerk CL controls the movements of the store clerk avatar A_CL and provides customer service, including interacting with customers.

[0106] FIG. 11 is a flowchart showing the flow of the specific customer determination process.

[0107] The chatbot control unit 303 starts bot response when the store clerk avatar A_CL is displayed.

[0108] The chatbot control unit 303 transmits images of all virtual stores and conversation log data of the store clerk avatars in each virtual store to the store clerk terminal HMDG1 (S201). The images of each virtual store and the conversation logs of the store clerk avatars are displayed on the store clerk terminal HMDG1 (S211).

[0109] The chatbot control unit 303 analyzes the conversation log data of the store clerk avatar in each virtual store and determines whether the customer who visited each virtual store is a specific customer who is likely to make a purchase (S202). In this example, customers who are likely to make a purchase are defined as specific customers, but customers who are difficult for the chatbot to handle are also detected as specific customers.

[0110] The chatbot control unit 303 returns to step S201 and repeats the process until a specific customer is detected (S202: No).

[0111] When the chatbot control unit 303 detects a specific customer (S202: Yes), it generates report data indicating in which virtual store the specific customer was detected and transmits the report data to the store clerk terminal HMDG1 (S203).

[0112] The store clerk terminal HMDG1 receives the notification data and displays the staff call sign (S212). The staff call sign is one form of notification display.

[0113] FIG. 12 is a diagram showing an example of a monitoring screen of the store clerk terminal.

[0114] 12, the monitoring screen 400 of the HMDG 1 displays images of the base store and the latest conversations between clerk avatars and customer avatars in all virtual stores. That is, the image of the base store is superimposed with the following: clerk avatar A_CL, first customer avatar A_CS1, and the latest conversations at the base store; clerk avatar A_CL, second customer avatar A_CS2, and the latest conversations at the first duplicate store; and clerk avatar A_CL, third customer avatar A_CS3, and the latest conversations at the second duplicate store. This allows clerk CL to simultaneously view the customer service situations at multiple virtual stores.

[0115] When a specific customer is detected at a base store, a staff call sign 401 is displayed in the conversation display area of ​​the base store on the monitoring screen 400 of the HMDG 1. The store name display area displayed at the top of each conversation display area functions as a bot changeover switch 402.

[0116] When the store clerk CL operates the bot changeover switch 402, the operation control mode of the store clerk avatar in the base store switches from automatic customer service mode to manual customer service mode. The communication control unit 309 mediates the transmission and reception of voice data between the store clerk terminal HMDG1 and the customer terminal HMDG2, enabling direct conversation between the store clerk and the customer. At that time, the chatbot control unit 303 causes the store clerk avatar A_CL to speak the store clerk's voice, thereby switching to real customer service with the store clerk CL without the customer being aware that they have switched from a chatbot to a real store clerk.

[0117] The chatbot conversation log is displayed on the monitoring screen 400 of the store clerk terminal HMDG1, so the store clerk can understand the process leading up to the switch to manual customer service mode and can hand over customer service from the store clerk avatar A_CL to him / her in a natural conversational flow.

[0118] Next, the specific customer determination process will be described in detail. Fig. 13 is a schematic diagram of a determination criteria table used in the specific customer determination process. Fig. 14 is a flowchart showing the specific customer determination process.

[0119] In the judgment criteria table 500 shown in FIG. 13, sequence numbers 501, conversation contents or keywords 502, and categories 503 classified by the weight of the judgment criteria are defined in association with each other. The conversation contents or keywords include recognition information. Category A is a conversation content and keyword that becomes a target for reporting if it appears twice in a conversation. Category B is a conversation content and keyword that becomes a target for reporting if it appears once in a conversation. The judgment criteria table 500 is stored in advance in the judgment criteria table storage unit 307.

[0120] As shown in FIG. 14, when the chatbot control unit 303 starts bot customer service in each virtual store, it initializes a counter value C set for each virtual store (S301).

[0121] The chatbot control unit 303 waits until the conversation log data is acquired (S302: No).

[0122] When the chatbot control unit 303 acquires the conversation log data (S302: Yes), it analyzes the conversation log data to extract the conversation content or keywords, and determines a category by referring to the criteria table 500 (S303).

[0123] If the chatbot control unit 303 determines that the conversation content or keywords correspond to category A (S303: A), it increments the counter value C by 2 (S304) and proceeds to step S307.

[0124] If the chatbot control unit 303 determines that the conversation content or keywords correspond to category B (S303: B), it increments the counter value C by 3 (S305) and proceeds to step S307.

[0125] If the chatbot control unit 303 determines that the conversation content or keywords do not correspond to category A or B (S303: not A or B), it increments the counter value C by 1 (S306) and proceeds to step S307.

[0126] After steps S304, S305, and S306, the chatbot control unit 303 determines whether the counter value C is 3 or greater.

[0127] If the counter value C is less than 3 (S307: No), the process returns to step S302.

[0128] If the counter value C is 3 or more (S307: Yes), report data including information capable of identifying the virtual store where the counter value C has become 3 or more is generated and transmitted to the store clerk terminal HMDG1 (S308).

[0129] According to this embodiment, customers can visit a virtual store where there are no other customers. Therefore, while it is difficult to rent out an entire store for shopping in the real world, according to this embodiment, customers can rent out an entire virtual store for online shopping.

[0130] On the other hand, even if the number of customers increases, the clerk terminal can display clerk avatars and customer avatars located in multiple virtual stores that the clerk is responsible for on a single screen, allowing the clerk to monitor the customer service situation in multiple virtual stores at once.

[0131] Furthermore, even if the number of customers increases, they are usually handled in an automated customer service mode using chatbot avatars, making it easy to increase the number of virtual stores staffed by real staff.

[0132] The chatbot control unit 303 then analyzes the conversation log data between the customer avatar and the store clerk avatar, and determines whether the customer is likely to make a purchase or whether the chatbot is unable to handle them as a specific customer, and reports this to the store clerk terminal, switching the store clerk avatar from automatic customer service mode to manual customer service mode. This allows the real store clerk to serve the specific customer through actual conversation, allowing them to be more actively involved in the purchase and promote dialogue with the customer, making it less likely that business opportunities will be missed.

[0133] According to this embodiment, unlike conventional online shops, the virtual store incorporates the brand's worldview and the expressions used in physical stores, creating an environment where customers can enjoy shopping while deepening their product knowledge through dialogue with specialized staff (avatars).

[0134] Second Embodiment In the second embodiment, a customer can select the level of customer service in a virtual store. Fig. 15 is a schematic diagram showing an example of a screen display displayed on a customer terminal.

[0135] The screen 600 shown in FIG. 15 is displayed on the customer terminals HMDG2 and HMDG3. When a customer logs in to the virtual store server 14 and visits the virtual store, the screen 600 is displayed. The screen 600 displays a desired customer service level questionnaire 601 on an image of the virtual store. The desired customer service level questionnaire 601 displays buttons 602 and 603 for selecting whether or not customer service is required. The chatbot control unit 303 causes the salesperson avatar A_CL to respond at a response frequency corresponding to the customer service level selected by the customer in the desired customer service questionnaire 601. In FIG. 15 , the customer service level can be selected in two stages. When the customer operates the button 603 for selecting "no customer service required," the chatbot control unit 303 temporarily suspends its function while remaining in the automatic customer service mode. This temporarily suspends the chatbot function. The salesperson avatar A_CL remains displayed on the screen 600 but does not speak to the customer, allowing the customer to leisurely view the product objects in the virtual store.

[0136] When the customer operates the button 602 to select whether or not assistance is required, the store clerk avatar A_CL starts up in auto-attendant mode and begins serving the customer through a chatbot.

[0137] According to this embodiment, the frequency of responses by the chatbot is adjusted based on the selection information of the customer service level entered by the customer, so that customer service control can be performed according to the customer's answers.

[0138] Third Embodiment In a third embodiment, when a store clerk avatar A_CL speaks in a chatbot, a synthesized voice generated based on the voice quality produced by the store clerk CL may be used.

[0139] Before the virtual store opens, the synthetic voice generation unit 305 collects the voice of the store clerk CL from the HMDG 1, generates synthetic voice information, and stores it in the synthetic voice information storage unit 308. Then, when the chatbot control unit 303 engages in a conversation, it causes the store clerk avatar A_CL to speak using the synthetic voice information.

[0140] If there are multiple store clerks, synthetic voice information is generated for each store clerk. Then, the login information of each store clerk is referenced, and the synthetic voice information generated from the voice of the store clerk monitoring the virtual store is used as the voice of the store clerk avatar A_CL of the virtual store that the logged-in store clerk is monitoring.

[0141] This prevents a sudden change in voice quality when the store clerk avatar A_CL switches from the automatic customer service mode to the manual customer service mode, thereby reducing the sense of discomfort caused by the mode switch.

[0142] Fourth Embodiment The fourth embodiment is an embodiment in which a customer terminal is able to select between pattern 1, in which the switching of the operation mode of the store clerk avatar A_CL is made visible, and pattern 2, in which the switching of the operation mode of the store clerk avatar A_CL is not made visible. The switching of the patterns may be configured to be performed from a setting screen of the store clerk terminal.

[0143] (Pattern 1) In pattern 1, the display form of the store clerk avatar A_CL changes between the automatic service mode and the manual service mode. This allows the customer to be notified that a store clerk has begun serving the customer when called from the customer terminal, thereby improving customer satisfaction.

[0144] As another example, a specialized staff avatar with more knowledge than the store clerk avatar A_CL may be prepared, and the store clerk avatar A_CL may be replaced by the specialized staff avatar.

[0145] Alternatively, using pattern 1, a pseudo-customer avatar may be displayed before a customer visits the virtual store, and a store clerk avatar A_CL may serve the pseudo-customer avatar in automatic customer service mode. When a customer avatar A_CS1 corresponding to an actual customer visits the store, the store clerk avatar A_CL may approach the customer avatar A_CS1 and begin serving the customer in automatic customer service mode, and switch to manual customer service mode when the customer is detected as a specific customer.

[0146] This allows customers who feel uneasy when they enter a virtual store and there are no other customers to feel at ease because of the presence of a pseudo-customer. At that time, the customer can look around the virtual store at their own pace without worrying about the store clerk avatar A_CL, as the store clerk avatar A_CL is serving the pseudo-customer. Then, when the customer is detected as a specific customer without realizing it, service begins in manual service mode, and the process can be smoothly carried out until the store clerk begins to actively assist the customer.

[0147] (Pattern 2) In pattern 2, the customer terminals HMDG2 and G3 do not clearly indicate whether the store clerk avatar A_CL is operating in the automatic service mode or the manual service mode.

[0148] This allows the customer to concentrate on shopping without being aware that the store clerk avatar A_CL has switched from the automatic customer service mode to the manual customer service mode.

[0149] Furthermore, the screen 600 of the store clerk terminal HMDG1 may be configured not to clearly indicate whether the automatic customer service mode or the manual customer service mode is being used. This prevents the screen 600 from looking unnatural by switching the display of the store clerk avatar A_CL between a chatbot and a real store clerk when there are many customers being served.

[0150] It should be noted that the present invention is not limited to the above-described embodiment and includes various modifications. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to an embodiment having all of the described configurations.

[0151] It is also possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment.It is also possible to add, delete, or replace part of the configuration of each embodiment with the configuration of another embodiment.

[0152] Furthermore, some or all of the above-described components and functions may be implemented in hardware by being designed, for example, with an integrated circuit, a general-purpose processor, or an application-specific processor. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. Furthermore, the above-described components and functions may be implemented by a program executed by the processor. Furthermore, the control lines and information lines shown are those considered necessary for explanation, and do not necessarily represent all control lines and information lines in the product. In reality, it may be assumed that almost all components are interconnected.

[0153] The above embodiment includes the following inventions. (Supplementary Note 1) A virtual reality processing server comprising: a processor; a communication device; and a memory, wherein the communication device is connected to a customer terminal operated by a customer visiting a virtual facility set up in a virtual space, and a staff terminal operated by a real staff member, respectively, and wherein the memory stores base facility information which is the original of the video information of the virtual facility, and wherein the processor, when the customer visits the virtual facility, reads out the base facility information and assigns a base facility to the customer, and transmits a video of a staff avatar placed in the base facility to the customer terminal, controls the operation of the staff avatar in an auto customer service mode which generates a response to the customer based on conversation log data between the customer and the staff avatar, determines based on the conversation log data whether the customer is a specific customer who requires an actual response from the staff member, and if it is determined that the customer is the specific customer, sends notification data to the staff terminal informing the staff member that the customer is the specific customer, and switches the operation control of the staff avatar to a manual customer service mode which causes the staff avatar to speak the voice of the staff member received from the staff terminal.

[0154] (Supplementary Note 2) A virtual reality processing method, comprising the steps of: when a customer visits a virtual facility set up in a virtual space, transmitting an image of a staff avatar placed in the virtual facility to a customer terminal operated by the customer; controlling the operation of the staff avatar in an auto-customer service mode that generates a response to the customer based on conversation log data between the customer and the staff avatar; determining based on the conversation log data whether the customer is a specific customer who requires a response from a real staff member; and, if it is determined that the customer is the specific customer, transmitting notification data to a staff terminal operated by the staff member informing them that the customer is the specific customer, and switching the operation control of the staff avatar to a manual customer service mode that causes the staff avatar to speak the voice of the staff member received from the staff terminal.

[0155] 1: Smartphone 2: Processor 3: Bus 4: Memory 5: Sensor device 6: Communication device 7: Video processing device 8: Audio processing device 9: Operation input device 10: Gaze detection device 10L: Left gaze detection sensor 10R: Right gaze detection sensor 11: Control device 13: Communication network 14: Virtual store server 41: Program storage area 42: Data storage area 43: Work area 51: GPS sensor 52: Geomagnetic sensor 53: Gyro sensor 54: Acceleration sensor 55: Attachment / detachment detection sensor 61: LAN communication device 62: Mobile wireless communication device 63: Close proximity wireless communication device 71: Camera 72L: Left display 72R: Right display 81: Microphone 82: Voice recognition unit 83: Decoder 84: Encoder 85L : Left speaker 85R : Right speaker 100 : Virtual store customer service system 201 : Processor 202 : RAM 203 : ROM 204 : HDD 205 : Input I / F 206 : Output I / F 207 : Communication I / F 208 : Bus 211 : Input device 212 : Output device 301 : Login processing unit 302 : Virtual store setting unit 303 : Chatbot control unit 304 : Customer avatar control unit 305 : Synthetic voice generation unit 306 : Base store information storage unit 307 : Judgment criteria table storage unit 308 : Synthetic voice information storage unit 309 : Communication control unit 400 : Monitoring screen 401 : Staff call sign 402 : Bot changeover switch 500 : Judgment criteria table 501 : Sequence number 502 : Conversation content or keyword 503: Category 600: Screen 601: Desired questionnaire 602: Button 603: Button A_CL: Clerk avatar A_CS1: First customer avatar A_CS2: Second customer avatar A_CS3: Third customer avatar CL: ClerkCS1: First customer CS2: Second customer CS3: Third customer G1: HMD G10: Housing G1a: HMD G2: HMD G3: HMD P1: User R1: Wireless router R2: Wireless router R3: Wireless router SH0: Base store SH1: First duplicate store SH2: Second duplicate store

Claims

1. A virtual reality processing server comprising a processor, a communication device, and a memory, the communication device being connected to a customer terminal operated by a customer visiting a virtual facility installed in a virtual space, and a staff terminal operated by real staff, the memory storing base facility information which is the original of video information of the virtual facility, the processor reading out the base facility information when the customer visits the virtual facility and assigning a base facility to the customer, and transmitting a video in which a staff avatar is placed in the base facility to the customer terminal, controlling the operation of the staff avatar in an auto customer service mode which generates a response to the customer based on conversation log data between the customer and the staff avatar, determining based on the conversation log data whether the customer is a specific customer who requires an actual response from the staff, and if it is determined that the customer is the specific customer, transmitting notification data to the staff terminal informing the staff that the customer is the specific customer, and switching operation control of the staff avatar to a manual customer service mode which causes the staff avatar to speak the voice of the staff received from the staff terminal.

2. A virtual reality processing server as described in claim 1, wherein the processor, when an additional customer occurs while the base facility is assigned to the customer, duplicates the base facility information to generate a duplicate facility, assigns the duplicate facility to the additional customer, places a staff avatar at each of the base facility and the duplicate facility, determines whether the customer and the additional customer are specific customers based on the conversation log data of the staff avatar placed at the base facility and the staff avatar placed at the duplicate facility, and switches the voice of the staff avatar placed at the base facility or the duplicate facility assigned to the customer or additional customer determined to be the specific customer to the voice received from the staff terminal.

3. A virtual reality processing server as described in claim 2, wherein the processor uses a synthetic voice using the voice quality of the staff member as the voice with which the staff avatar speaks the conversation generated by the processor in the automatic customer service mode.

4. A virtual reality processing server as described in claim 1, wherein the processor notifies the customer terminal that the operation mode of the staff avatar has switched from the automatic customer service mode to the manual customer service mode.

5. A virtual reality processing server as described in claim 1, wherein the processor transmits selection information to the customer terminal to allow the customer to select a level of customer service desired by the customer, and adjusts the frequency of responses to the customer by the staff avatar in the auto customer service mode based on the selection information.

6. A virtual reality processing server as described in claim 2, wherein the processor transmits to the staff terminal a monitoring screen on which images of the staff avatar and a customer avatar representing the customer placed in the base facility and images of the staff avatar and an additional customer avatar representing the additional customer placed in the replicated facility are arranged on a single screen, displays on the monitoring screen a conversation log with the customer and a conversation log with the additional customer, and displays on the monitoring screen a notification display based on the report data.

7. A virtual reality processing method, comprising the steps of: when a customer visits a virtual facility set up in a virtual space, transmitting an image in which a staff avatar is placed in the virtual facility to a customer terminal operated by the customer; controlling the operation of the staff avatar in an auto-service mode in which a response to the customer is generated based on conversation log data between the customer and the staff avatar; determining based on the conversation log data whether the customer is a specific customer who requires a response from an actual staff member; and, if it is determined that the customer is the specific customer, transmitting notification data to a staff terminal operated by the staff member informing them that the customer is the specific customer, and switching control of the operation of the staff avatar to a manual service mode in which the staff avatar speaks the voice of the staff member received from the staff terminal.

Images