Avatar control system, avatar control method, and program

Abstract

To enable a user existing in a real space corresponding to a virtual space and a user existing in another real space different from the real space corresponding to the virtual space to communicate smoothly through the avatar of each user in the virtual space.SOLUTION: An avatar control system comprises: a virtual space providing unit for outputting, on a user terminal, a virtual space that simulates a first real space of a predetermined range; a sensor for detecting a first user who exists as a person belonging to the first real space; and an avatar control unit for identifying a real position in the first real space of the first user detected by the sensor, placing a first avatar corresponding to the first user at a position in the virtual space corresponding to the real position, and controlling communication between the first avatar and a second avatar placed in the virtual space corresponding to a second user who exists in a second real space other than the first real space.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an avatar control system, an avatar control method, and a program. [Background technology]

[0002] There is known a technique for displaying an image of real space captured by a camera on a head mounted display (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2021-105763 A Summary of the Invention [Problem to be solved by the invention]

[0004] When making an avatar act in a virtual space such as a metaverse that imitates a certain range of real space or a digital twin that is constructed corresponding to a certain range of real space, avatars of users with different real space environments may exist together, such as a user who exists in a real space that corresponds to the virtual space and a user who exists in another real space different from the real space that corresponds to the virtual space. Taking such a situation into consideration, it is preferable to facilitate communication between users using avatars in the virtual space.

[0005] The present invention aims to enable smooth communication between a user existing in a real space corresponding to a virtual space and a user existing in another real space different from the real space corresponding to the virtual space, through each user's avatar in the virtual space. [Means for solving the problem]

[0006] One aspect of the present invention that solves the above-mentioned problems is an avatar control system that includes a virtual space providing unit that outputs a virtual space simulating a first real space of a predetermined range on a user terminal, a sensor that detects a first user who exists as someone belonging to the first real space, and an avatar control unit that identifies a real position in the first real space of the first user detected by the sensor, causes a first avatar corresponding to the first user to exist at a position in the virtual space corresponding to the real position, and controls communication between the first avatar and a second avatar caused to exist in the virtual space corresponding to a second user who exists in a second real space other than the first real space.

[0007] One aspect of the present invention is an avatar control method in an avatar control system, the avatar control method including: a virtual space providing step in which a virtual space providing unit outputs a virtual space simulating a first real space of a predetermined range on a user terminal; and an avatar control step in which an avatar control unit identifies a real position in the first real space of a first user detected by a sensor that detects a first user existing as someone belonging to the first real space, causes a first avatar corresponding to the first user to exist at a position in the virtual space corresponding to the real position, and controls communication between the first avatar and a second avatar caused to exist in the virtual space corresponding to a second user existing in a second real space other than the first real space.

[0008] One aspect of the present invention is a program for causing a computer in an avatar control system to function as a virtual space providing unit that outputs a virtual space simulating a predetermined range of a first real space on a user terminal, an avatar control unit that identifies the real position in the first real space of a first user detected by a sensor that detects a first user existing as someone belonging to the first real space, causes a first avatar corresponding to the first user to exist at a position in the virtual space corresponding to the real position, and controls communication between the first avatar and a second avatar caused to exist in the virtual space corresponding to a second user existing in a second real space other than the first real space. Effect of the Invention

[0009] According to the present invention, a user existing in a real space corresponding to a virtual space and a user existing in another real space different from the real space corresponding to the virtual space can communicate smoothly through each user's avatar in the virtual space. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an example of the overall configuration of an avatar control system according to an embodiment of the present invention. [Diagram 2] 2 is a diagram illustrating an example of a functional configuration of a metaverse management server in the present embodiment. FIG. [Diagram 3] FIG. 4 is a diagram showing an example of store clerk user information in the present embodiment. [Figure 4] 5 is a diagram showing an example of captured image position information in the present embodiment. FIG. [Diagram 5] 4 is a diagram showing an example of a processing procedure executed by the avatar control system in the present embodiment in relation to avatar control. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] <Embodiment> An example of the overall configuration of an avatar control system in this embodiment will be described with reference to Fig. 1. The avatar control system of this embodiment causes an avatar corresponding to a user U (Ua, Ub) to exist in a virtual space that mimics a facility FC in real space. In this embodiment, the virtual space simulating the facility FC may be a three-dimensional space that faithfully reproduces the facility FC as a metaverse or digital twin environment. While reproducing the structure and shape of the facility FC and the positions of installations and obstacles in the facility FC, the virtual space simulating the facility FC may have the installations and obstacles replaced with objects different from reality, such as trees and rocks, or the patterns and textures of the walls, floors, etc. may be different from reality. The virtual space simulating the facility FC may also be constructed in two dimensions. A user U can operate a user terminal (smart glasses 100, user terminal 300) owned by the user U to make the corresponding avatar act in the virtual space. The action of the avatar in response to the operation here may be movement in the virtual space, a change in posture, a gesture, a hand gesture, a change in facial expression, speech, etc. In addition, the user U can communicate with other users by making the avatar corresponding to the user U communicate with the avatar of the other user U in the virtual space.

[0012] The environment of the facility FC is not particularly limited, but examples include commercial facilities such as shopping malls and department stores, exercise facilities such as training gyms and gymnasiums, entertainment facilities such as amusement parks, manufacturing sites, etc. In the following explanation, an example is given in which the facility FC is a commercial facility. In this case, the virtual space modeled after the facility FC includes the commercial facilities owned by the facility FC. In this case, the user Ua is a clerk working at the commercial facility serving as the facility FC. In this respect, the user Ua is treated as belonging to the facility FC. Meanwhile, the user Ub, who exists in the real space serving as the out-of-facility environment OFC, is a customer who communicates with the user Ua through contact (communication) between his / her own avatar and an avatar corresponding to the user Ua in the virtual space of the facility FC.

[0013] In the facility FC shown in the figure, a plurality of cameras 10 are provided at predetermined positions. Each of the cameras 10 is provided so as to capture an image of a predetermined spatial range in the facility FC. Images captured by the cameras 10 are used to track a user Ua who actually exists in the facility FC. The number of cameras 10 provided in the facility FC is not particularly limited, and it is sufficient that the number of cameras 10 necessary for tracking the user Ua in the facility FC is provided at predetermined positions. Each of the cameras 10 is connected to a virtual space management server 200 (an example of an avatar control device) via a network, for example, from a gateway 20. The cameras 10 transmit captured images to the virtual space management server 200. The virtual space management server 200 may detect the position of the user Ua in the facility FC, which changes in accordance with the movement of the user Ua, based on the captured images.

[0014] Among the multiple cameras 10, the camera 10-E is provided to capture an image of the user Ua entering the facility FC through the entrance ENT. In the present embodiment, when the user Ua is a store clerk working in the facility FC, the entrance ENT may be reserved for employees only. The captured images captured by the camera 10-E are used by the virtual space management server 200 to generate an avatar of the user Ua entering the facility FC. Note that a plurality of cameras 10-E for obtaining captured images used to generate the avatar of the user Ua may be disposed so that the user Ua entering through the entrance ENT can be tracked at a predetermined distance. Hereinafter, the camera 10-E will also be referred to as the entrance camera 10-E to distinguish it from the other cameras 10.

[0015] In the facility FC, the user Ua wears smart glasses 100. The smart glasses 100 are terminals (user terminals) through which the user Ua operates an avatar corresponding to the user Ua while viewing a virtual space corresponding to the facility FC. The smart glasses 100 are communicably connected to the virtual space management server 200 via a network. The user Ua can use the smart glasses 100 to make the avatar corresponding to the user Ua communicate with avatars corresponding to other users present in the virtual space. Note that some of the users Ua may not wear the smart glasses 100 and therefore may not communicate in the virtual space.

[0016] Since the user terminal used by the user Ua is a wearable terminal such as smart glasses 100, the user Ua, who is a store clerk, can easily communicate with customers visiting the facility FC and with the user Ub in the external environment OFC via the virtual space, even while moving around within the facility FC or performing manual tasks. In addition, when the degree of freedom regarding movement and work as described above does not need to be considered, a signage provided at a predetermined position in the facility FC and having a predetermined input interface may be used as the user terminal. Also, for example, a tablet terminal, a smartphone, or the like may be used as the user terminal.

[0017] Each user Ub present in the out-of-facility environment OFC has a user terminal 300. The user terminal 300 is communicably connected to the virtual space management server 200. By operating the user terminal 300, the user Ub can make his / her avatar act in the virtual space corresponding to the facility FC.

[0018] For example, smart glasses 100 used within a facility FC may display a virtual space that mimics the facility FC in three dimensions, while a user terminal 300 used in an outside facility environment OFC may display a virtual space that mimics the facility FC in a simplified two-dimensional manner.

[0019] The virtual space management server 200 (an example of an avatar control device) performs management related to the virtual space simulating the facility FC. The virtual space management server 200 recognizes an image portion of the user Ua who entered through the entrance ENT from the captured image obtained by the entrance camera 10-E, and generates an avatar corresponding to the user Ua using the recognized image portion of the user Ua. In this case, the virtual space management server 200 may generate a realistic three-dimensional avatar (real avatar) that makes use of the realism of the recognized image portion of the user Ua, may generate a three-dimensional or two-dimensional character-like avatar having the characteristics of the user Ua, or may generate a character-like avatar having an aspect unrelated to the characteristics of the user Ua.

[0020] In addition, the virtual space management server 200 generates an avatar of the corresponding user Ub by using the captured image of the user Ub transmitted from the user terminal 300. The avatar of the user Ub may also be generated as a real avatar or as a character-like avatar.

[0021] In the following description, the user Ua present in the facility FC will also be referred to as a clerk user Ua, and the user Ub present in the outside facility environment OFC will also be referred to as a customer user Ub. The avatar generated corresponding to the clerk user Ua will also be referred to as a clerk avatar (an example of a first avatar), and the avatar generated corresponding to the customer user Ub will also be referred to as a customer avatar (an example of a second avatar).

[0022] The virtual space management server 200 provides a virtual space that imitates the facility FC. The virtual space management server 200 also tracks the position (an example of an actual position) of the store clerk user Ua who belongs to the facility FC by using the captured image received from the camera 10. In other words, the virtual space management server 200 constantly identifies the position of the store clerk user Ua in the facility FC. Then, the virtual space management server 200 moves the tracked store clerk user Ua within the virtual space so that a corresponding store clerk avatar is present at the position of the tracked store clerk user Ua. Furthermore, the virtual space managing server 200 moves the corresponding avatar in response to an operation of the user terminal 300 by the customer user Ub. In addition, the virtual space management server 200 may control a corresponding store clerk avatar to speak in response to input of the voice of the store clerk user Ua to the microphone equipped in the smart glasses 100. Similarly, the virtual space management server 200 may control a corresponding customer avatar to speak in response to input of the voice of the customer user Ub to the user terminal 300 or to a character input operation corresponding to the voice. The virtual space management server 200 controls the actions of the clerk avatar corresponding to the clerk user Ua and the customer avatar corresponding to the customer user Ub in this way, thereby enabling communication between the clerk avatar and the customer avatar in the virtual space. As a result, communication can be achieved between the clerk user Ua, who exists in the real space as the facility FC, and the customer user Ub, who exists in the real space as the outside facility environment OFC.

[0023] An example of the functional configuration of the virtual space management server 200 will be described with reference to Fig. 2. The virtual space management server 200 includes, as hardware, a central processing unit (CPU), read only memory (ROM), random access memory (RAM), and storage devices such as hard disk drives (HDDs) and solid state drives (SSDs). The function of the investigation device shown in the figure is realized by a CPU included in the hardware of the virtual space management server 200 executing a program.

[0024] The virtual space management server 200 in the figure includes a communication unit 201, a control unit 202, and a storage unit 203. The communication unit 201 communicates with the camera 10, the smart glasses 100, and the user terminal 300 via a network. The control unit 202 executes various controls in the virtual space management server 200. The control unit 202 includes an avatar generation unit 221, a virtual space providing unit 222, and an avatar control unit 223.

[0025] The avatar generation unit 221 generates an avatar. The avatar generation unit 221 generates a store clerk avatar by using an image portion of a person as the store clerk user Ua recognized from a captured image transmitted from the entrance camera 10-E of the facility FC. The avatar generation unit 221 also generates a customer avatar by using an image portion of a person recognized from an image transmitted from the user terminal 300.

[0026] The virtual space providing unit 222 provides a virtual space that imitates the facility FC. That is, the virtual space providing unit 222 generates an environment as a virtual space that imitates the facility FC. The virtual space providing unit 222 causes the avatar generated by the avatar generating unit 221 to exist in the virtual space. The virtual space providing unit 222 causes the environment of the virtual space to be reproduced by display and sound on the smart glasses 100 and the user terminal 300.

[0027] The avatar control unit 223 executes control regarding the behavior of avatars (store clerk avatars, customer avatars) existing in the virtual space. Here, the behavior of the avatars controlled by the avatar control unit 223 may be, as described above, movement in the virtual space, changes in posture, gestures, hand movements, changes in facial expression, speech, and the like.

[0028] The storage unit 203 stores various types of information corresponding to the virtual space management server 200. The storage unit 203 includes a clerk user information storage unit 231, a customer user information storage unit 232, a captured image position information storage unit 233, and a virtual space data storage unit 234.

[0029] The clerk user information storage unit 231 stores user information (clerk user information) for each clerk user Ua. 3 shows an example of the structure of clerk user information corresponding to one clerk user Ua. The clerk user information in the figure includes fields for a user ID, user authentication information, and terminal use information. The user ID field stores a user ID that uniquely identifies the corresponding store clerk user Ua. The user authentication information field is information used for user authentication performed in the avatar control system. The user authentication information may be, for example, a password associated with a user ID, or an image of the face of the corresponding user, an iris, a fingerprint, or the like. The terminal usage information is information related to the use of the smart glasses 100 by the corresponding store clerk user Ua. Specifically, the terminal usage information indicates whether the store clerk user Ua wears the smart glasses 100 in the facility FC and uses them in the facility FC. In addition, when the corresponding store clerk user Ua uses the smart glasses 100, the terminal usage information indicates the terminal ID of the corresponding smart glasses 100. The store clerk user information may also include additional information such as the tenant (store) to which the corresponding store clerk user Ua belongs.

[0030] Returning to Fig. 2 for the explanation, the customer user information storage unit 232 stores user information (customer user information) for each customer user Ub registered as a user of the virtual space. The customer user information corresponding to one customer user Ub may include, for example, a user ID and user authentication information. The user authentication information corresponding to a customer user Ub is information used to log in to the virtual space, and may be a password associated with a user ID, as with the store clerk user information, or may be an image of the corresponding user's face, an iris, a fingerprint, etc.

[0031] The captured image position information storage unit 233 stores captured image position information. The captured image position information is information that associates a position (pixel) in an image captured by each camera 10 with a position (coordinates) in the facility FC. Fig. 4 shows an example of captured image position information stored in the captured image position information storage unit 233. The captured image position information in the figure has a structure in which in-image position information is stored in association with each camera ID of the camera 10. The in-image position information is information in which each pixel of a captured image captured by the corresponding camera 10 is associated with the coordinates of a spatial portion of the facility FC included in the same captured image. In addition, in the captured image position information, a camera installation position indicating a position in the facility FC where the camera 10 is installed may be further stored in association with the camera ID. Alternatively, camera installation position information in which the camera installation position is associated with each camera 10 may be stored in the storage unit 203. When the tracking target user is no longer recognized from a captured image due to the user's movement in the facility FC, for example, the camera installation position information may be used to identify a captured image that newly includes the tracking target user.

[0032] Returning to the explanation in Fig. 2, the virtual space data storage unit 234 stores data (virtual space data) for constructing a virtual space simulating a facility FC. The virtual space providing unit 222 constructs a virtual space using the virtual space data stored in the virtual space data storage unit 234. The virtual space data also includes information associating coordinates in the virtual space (virtual space coordinates) with coordinates in the space of the corresponding facility FC (facility coordinates).

[0033] An example of a process performed by the avatar control system in relation to controlling an avatar will be described with reference to the sequence diagram of FIG. First, a process related to avatar control of a store clerk avatar corresponding to a store clerk user Ua existing in the real space as the facility FC will be described. Step S100: In the facility FC, the entrance camera 10-E transmits captured images to the virtual space management server 200, for example, constantly.

[0034] Step S102: In the virtual space management server 200, the avatar generation unit 221 recognizes, from the received captured image, a person who is about to enter the facility FC from the entrance ENT. The entrance camera 10-E may be configured to constantly capture images while detecting a person in the captured images, and when a person is detected, the captured image including the detected person may be transmitted. In this case, the process of step S102 may be omitted.

[0035] Step S104: The virtual space providing unit 222 performs authentication to determine whether or not the person recognized in step S102 is a legitimate store clerk user Ua. For example, the virtual space providing unit 222 may perform authentication based on whether or not a photo of the same face as the person recognized in step S102 is searched for among the store clerk user information stored in the store clerk user information storage unit 231. Alternatively, the virtual space providing unit 222 may perform authentication by comparing a user ID, iris, fingerprint, or the like that the store clerk user Ua causes an authentication sensor to read when entering through the entrance ENT with authentication information for each store clerk user Ua stored in the store clerk user information storage unit 231.

[0036] In addition, the smart glasses 100 are not necessarily uniquely associated with the store clerk user Ua and managed, and the store clerk user Ua may be allowed to use different smart glasses 100 at any time. In this case, the store clerk user Ua who enters through the entrance ENT may perform an operation on the smart glasses 100 to associate the smart glasses 100 to be used this time with his / her own user ID on the smart glasses 100. In this case, the smart glasses 100 may transmit a terminal use notification including the associated user ID and his / her own terminal ID to the virtual space management server 200. The virtual space management server 200 may store the terminal ID included in the received terminal use notification in the terminal use information in the store clerk user information that stores the same user ID as that included in the received terminal use notification. Alternatively, a position estimation system that estimates a position may be provided for each pair of smart glasses 100, and the smart glasses 100 whose estimated position matches that of the store clerk user Ua tracked by the camera 10 may be associated with the user ID of the store clerk user Ua. The position estimation system that estimates a position for each pair of smart glasses 100 may be, for example, a position estimation system that estimates the position of a first user as described below.

[0037] Step S106: The avatar generation unit 221 generates a store clerk avatar that imitates the person recognized from the captured image in step S102, using the image portion of the person. The avatar control unit 223 manages the generated store clerk avatar in association with the store clerk user Ua authenticated in step S104. In addition, the avatar generation unit 221 may generate a corresponding store clerk avatar every time the corresponding store clerk user Ua enters the facility FC, and the avatar control unit 223 may manage the generated store clerk avatar in association with the store clerk user Ua. Alternatively, information in which the corresponding store clerk avatar and the store clerk user Ua are associated with each other may be stored in the storage unit 203 in response to the store clerk user Ua entering the facility FC once. In this case, the corresponding store clerk avatar stored in the storage unit 203 may be made to exist in the virtual space in response to the store clerk user Ua subsequently entering the facility FC. In other words, in this case, it is not necessary to generate a corresponding store clerk avatar every time the store clerk user Ua enters the facility FC.

[0038] As described above, the clerk avatar generated in step S106 may or may not be used by the corresponding clerk user Ua when the clerk user Ua is using the smart glasses 100. When the clerk user Ua is using the smart glasses 100, the clerk user Ua can operate the corresponding clerk avatar, but when the clerk user Ua is not using the smart glasses 100, the clerk user Ua cannot operate the corresponding clerk avatar. In other words, the clerk user Ua who is using the smart glasses 100 can communicate with the customer user Ub through the virtual space, but the clerk user Ua who is not using the smart glasses 100 cannot communicate with the customer user Ub through the virtual space. From the customer user Ub's perspective, it is preferable to know whether the corresponding clerk user Ua can serve the clerk avatar in the virtual space. Therefore, the avatar generation unit 221 or the avatar control unit 223 may indicate whether or not the corresponding clerk avatar can serve (communicate with the customer avatar) in the virtual space (i.e., whether or not the corresponding clerk user Ua is using the smart glasses 100) in a predetermined manner. Whether or not the clerk avatar can serve the customer may be identified based on whether the terminal usage information in the clerk user information of the corresponding clerk user indicates whether the smart glasses 100 are used or not. Specifically, as a mode of indicating whether or not a customer can be served, for example, a predetermined sign (service availability sign) indicating that a customer can be served may be attached to a store clerk avatar that can serve customers. Such a sign may be, for example, a medal or badge that the store clerk avatar wears, or may be attached as letters or symbols in the vicinity of the store clerk avatar. In addition, the color tone or brightness of the avatar body may be different between a store clerk avatar that can serve customers and a store clerk avatar that cannot serve customers. In addition, a user who is a customer visiting the facility FC in the real space may also wear the smart glasses 100 and be able to act in the facility FC while experiencing the virtual space in which an avatar corresponding to the user exists. In this case, in the virtual space of the facility FC, in addition to the store clerk avatar, a customer avatar of a customer who visited the facility FC exists as an avatar belonging to the facility FC. Therefore, in this case, a mark or the like may be attached in a predetermined manner so that the store clerk avatar and the customer avatar of the customer who visited the facility FC can be distinguished from each other as the avatars belonging to the facility FC.

[0039] Step S108: Each of the cameras 10 in the facility FC transmits the captured images obtained by capturing images to the virtual space management server 200. The transmission of captured images by the cameras 10 may be performed constantly.

[0040] Step S110: In the virtual space management server 200, the avatar control unit 223 uses the captured images transmitted from each of the cameras 10 to execute control over the behavior of the store clerk avatar generated in step S106 (behavior control). The behavior control in step S110 includes control (movement control) regarding the movement of the store clerk avatar in the virtual space. In controlling the movement of the store clerk avatar, the avatar control unit 223 identifies the store clerk user Ua recognized in step S102 from the captured images transmitted from each of the cameras 10. At this time, the avatar control unit 223 identifies, among the people recognized from the captured images, a person having a feature amount matching the store clerk user Ua recognized in step S102 as the same store clerk user Ua recognized in step S102. The avatar control unit 223 controls the clerk avatar corresponding to the identified clerk user Ua to exist at a position (coordinates) in the virtual space corresponding to the position (intra-facility coordinates) where the identified clerk user Ua exists in the facility FC. That is, the avatar control unit 223 refers to the captured image position information to identify the intra-facility coordinates where the identified clerk user Ua exists in the facility FC. The avatar control unit 223 causes the corresponding clerk avatar to exist so as to be positioned at the intra-virtual space coordinates corresponding to the identified intra-facility coordinates. At this time, the avatar control unit 223 controls the corresponding clerk avatar existing in the virtual space to be positioned at the intra-virtual space coordinates corresponding to the changed intra-facility coordinates in response to the intra-facility coordinates where the clerk user Ua is located being changed due to the movement of the clerk user Ua in the facility FC. That is, the clerk avatar is made to move in the virtual space so as to reflect the movement of the corresponding clerk user Ua in the facility FC which is the real space.

[0041] In addition, as the behavior control in step S110, the avatar control unit 223 may detect the gestures (body movements, hand movements) and facial expressions of the corresponding store clerk avatar from the captured images transmitted from each of the cameras 10, and control the corresponding store clerk avatar to reflect the detected gestures and facial expressions. The processes in steps S108 and S110 may be loop processes that are regularly executed.

[0042] Step S112: A clerk user Ua who uses the smart glasses 100 in the facility FC wears corresponding smart glasses 100. The clerk user Ua who wears the smart glasses 100 can operate a clerk avatar corresponding to the clerk user Ua who exists in the virtual space while viewing the virtual space of the facility FC displayed on the smart glasses 100 while existing in the facility FC which is a real space. The smart glasses 100 transmit operation information according to the operation performed by the clerk user Ua to the virtual space management server 200. Here, the operation of the clerk avatar by the clerk user Ua may include, for example, detection of the line of sight of the clerk user Ua (for example, the horizontal direction (azimuth) and the elevation / depression angle) by the smart glasses 100. The smart glasses 100 may transmit information indicating the detected line of sight of the clerk avatar as avatar operation information indicating the line of sight of the corresponding clerk avatar. In addition, the operation of the clerk avatar by the clerk user Ua may include the clerk user Ua speaking to have a conversation with another avatar corresponding to the clerk user Ua existing in the virtual space. The smart glasses 100 include a microphone. The smart glasses 100 may collect the voice of the clerk user Ua by the microphone and transmit the collected voice signal as avatar operation information. The avatar operation information includes the user ID of the corresponding store clerk user Ua or the terminal ID of the smart glasses 100 that are the transmission source.

[0043] Step S114: In the virtual space management server 200, the avatar control unit 223 controls the behavior of the corresponding store clerk avatar in accordance with the avatar operation information transmitted in step S112. For example, when the received avatar operation information includes a line of sight direction, avatar control unit 223 may perform control so that the corresponding store clerk avatar looks in the same direction in the virtual space as the line of sight direction indicated in the avatar operation information. Furthermore, when the received avatar operation information includes a voice signal, the avatar control unit 223 may control the corresponding store clerk avatar to speak in the virtual space using the voice signal included in the avatar operation information. The processing in steps S108 and S110 may be a loop processing that is repeatedly executed every time an operation is performed by the store clerk user Ua.

[0044] Next, a process related to avatar control of a customer avatar corresponding to a customer user Ub existing in a real space as the outside-facility environment OFC will be described. Step S120: When using the virtual space, the customer user Ub operates the user terminal 300 owned by the customer user Ub to transmit a login request for logging in to the virtual space to the virtual space management server 200. The login request includes user authentication information including the user ID of the corresponding customer user Ub and a password, etc.

[0045] Step S122: In the virtual space management server 200, the virtual space providing unit 222 executes an authentication process corresponding to the user authentication information included in the login request received in response to step S120. Specifically, the virtual space providing unit 222 may determine whether or not authentication has been established based on a result of comparing the user authentication information stored in the customer user information stored in the customer user information storage unit 232 with the user authentication information included in the login request.

[0046] Step S124: If the authentication in step S122 is successful, the customer user Ub operates the user terminal 300 to cause the user terminal 300 to transmit a captured image of himself or herself to the virtual space management server 200.

[0047] Step S126: In the virtual space management server 200, the avatar generation unit 221 uses the captured image transmitted in step S124 to generate a customer avatar that resembles the corresponding customer user Ub. The avatar control unit 223 manages the generated customer avatar in association with the customer user Ub authenticated in step S122. The avatar control unit 223 causes the generated customer avatar to appear in the virtual space. The customer avatar generated as described above and associated with the customer user Ub may be stored in the storage unit 203. In this case, when the customer user Ub subsequently uses the virtual space, the customer avatar stored in the storage unit 203 can be made to appear in the virtual space even if the avatar generation unit 221 does not generate a customer avatar.

[0048] Step S128: A space within the virtual space is displayed on the user terminal 300 of the customer user Ub. On the user terminal 300, the virtual space of a spatial range observed from the viewpoint of a customer avatar corresponding to the customer user Ub may be displayed. The customer user Ub can operate the user terminal 300 to change the viewpoint of the customer avatar corresponding to the customer user Ub, move the customer avatar in the virtual space, and make the customer avatar speak. In response to an operation performed by the customer user Ub to cause the customer avatar to act in this way, the user terminal 300 transmits avatar operation information to the virtual space management server 200. The avatar operation information may include, for example, the user ID of the corresponding customer user.

[0049] Step S130: In the virtual space management server 200, the avatar control unit 223 controls the behavior of the corresponding customer avatar based on the avatar operation information received in response to step S128. In this manner, the behavior of the customer avatar is controlled by the avatar control unit 223, so that the customer avatar acts in the virtual space according to the intention of the corresponding customer user Ub.

[0050] In a virtual space environment in which a clerk avatar corresponding to each clerk user Ua and a customer avatar corresponding to each customer user Ub act as described above, the clerk user Ua and the customer user Ub can communicate with each other as follows. That is, from the viewpoint of the store clerk user Ua, a customer avatar who visits the facility in the virtual space can be served by a store clerk avatar corresponding to the user, and for example, products similar to those sold at the facility FC can be sold in the virtual space. Also, the store clerk user Ua can provide services such as product sales to customers who actually visit the facility FC. Furthermore, customer user Ub can make a transaction as if purchasing a product sold at facility FC while communicating with store clerk user Ua who is present at facility FC as a physical store, by having his or her avatar, which is his or her avatar in the virtual space, communicate with store clerk avatar without actually going to facility FC in the real space. In this manner, in this embodiment, the store clerk user Ua and the customer user Ub can smoothly communicate with each other through the virtual space.

[0051] At this time, the customer user Ub may determine which store clerk avatar is available to serve the customer based on the service availability indicators attached to the store clerk avatars existing in the virtual space described above, and may select the store clerk avatar that is available to serve the customer and operate it so that his or her own customer avatar communicates with the customer. In addition, in the virtual space, information about the product may be presented that would be difficult to realize in the facility FC in the real space, such as a video of the product actually being used or a video of the product being disassembled. In a transaction in a virtual space, for example, a customer user Ub may make a payment using various online payment services, and the purchased goods may be delivered to the customer user Ub from an EC (Electronic Commerce) system corresponding to the facility FC.

[0052] In addition, the virtual space providing unit 222 may perform control so that the movement corresponding to the buying and selling of products in the facility FC in the real space is also reflected in the virtual space. As an example, by managing individual products in a store in the facility FC in the real space using electronic shelf labels, product management can be performed in the corresponding store in the virtual space so that the inventory of products in the store decreases in response to the purchase of products by customers.

[0053] Depending on the operation of the virtual space, for example, authentication of at least one of the store clerk user Ua and the customer user Ub may not be required.

[0054] As mentioned above, the environment of the facility FC may be other than a commercial facility, for example, a sports facility, an amusement park, a manufacturing site, etc. As an example, when the facility FC is an exercise facility, the first user in the facility FC may be an instructor, and the second user in the outside facility environment OFC may be a user who receives instruction from the instructor. In this case, the first user and the second user communicate with each other in the virtual space using an instructor avatar and a user avatar, so that the second user can train under the instruction of the instructor as the first user. Also, when the facility FC is a manufacturing site, a first user as a worker in the facility FC can receive instructions from a second user as a factory manager in the outside facility environment OFC through the virtual space. In this case, the factory manager can grasp the state of the worker by observing the state in the virtual space while moving the second avatar corresponding to the first user by operating the user terminal 300. In this case, for example, the virtual space management server 200 may store history information tracking each first user as a worker in the virtual space. The history stored in this way can be used as a consideration material for improving the work efficiency at the manufacturing site as the facility FC.

[0055] It should be noted that the sensor for capturing the first user in a traceable manner in the facility FC is not limited to the camera 10. As an example, a position estimation system capable of estimating the position of the first user based on the arrival time difference between multiple receivers of radio waves such as UWB (Ultra-Wide Band) transmitted from a transmitter carried by the first user may be applied. In addition, a sensor that captures the first user in a facility FC in a trackable manner may be implemented by having the first user carry a transmitter as a BLE (Bluetooth Low Energy) beacon, providing receivers at various locations within the facility FC to receive data transmitted from the transmitter, and applying a position estimation system that can estimate the position of the first user based on the position of the receiver that received the data. Furthermore, a plurality of configurations as sensors for tracking and capturing such a first user may be used in combination.

[0056] For example, a plurality of devices on a network, each of which is assigned a specific function, may cooperate to realize the function of the virtual space management server 200 in FIG.

[0057] In addition, a program for realizing the functions of the above-mentioned camera 10, smart glasses 100, virtual space management server 200, and user terminal 300 may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to perform processing as the above-mentioned smart glasses 100, virtual space management server 200, and user terminal 300. Here, "reading a program recorded on a recording medium into a computer system and executing it" includes installing the program into a computer system. The "computer system" here includes hardware such as an OS and peripheral devices. In addition, the "computer system" may include multiple computer devices connected via a network including the Internet, a WAN, a LAN, a communication line such as a dedicated line, etc. In addition, the computer system may be a classical computer, or all or part of the functions may be realized by a quantum computer. In addition, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, an optical magnetic disk, a ROM, a CD-ROM, etc., and a storage device such as an HDD or SSD built into a computer system. In this way, the recording medium storing the program may be a non-transient recording medium such as a CD-ROM. The recording medium also includes a recording medium installed inside or outside the distribution server and accessible from the distribution server in order to distribute the program. The code of the program stored in the distribution server's recording medium may be different from the code of the program in a format executable by the terminal device. In other words, the format in which the program is stored in the distribution server does not matter as long as it can be downloaded from the distribution server and installed in a format executable by the terminal device. Note that the program may be divided into multiple parts, each of which may be downloaded at a different time and then combined in the terminal device, or each of the divided programs may be distributed by a different distribution server. Furthermore, the "computer-readable recording medium" includes a memory that holds the program for a certain period of time, such as a volatile memory (RAM) in a computer system that is a server or client when the program is transmitted via a network.The program may be for implementing some of the above-mentioned functions, or may be a so-called differential file (differential program) that can implement the above-mentioned functions in combination with a program already recorded in the computer system.

[0058] <Additional Notes> (1) One aspect of this embodiment is an avatar control system that includes a virtual space providing unit that outputs a virtual space simulating a first real space of a predetermined range on a user terminal, a sensor that detects a first user who exists as someone belonging to the first real space, and an avatar control unit that identifies a real position in the first real space of the first user detected by the sensor, causes a first avatar corresponding to the first user to exist at a position in the virtual space corresponding to the real position, and controls communication between the first avatar and a second avatar caused to exist in the virtual space corresponding to a second user who exists in a second real space other than the first real space.

[0059] (2) One aspect of this embodiment is the avatar control system described in (1), wherein the avatar control unit controls a corresponding first avatar to act in the virtual space in response to a predetermined operation performed by the first user on a first user terminal in the first real space, and controls a corresponding second avatar to act in the virtual space in response to a predetermined operation performed by the second user on a second user terminal in the second real space.

[0060] (3) One aspect of the present embodiment is the avatar control system described in (2), in which the first user terminal may be a wearable terminal that can be worn by the first user.

[0061] (4) One aspect of this embodiment is an avatar control system as described in (2) or (3), in which the avatar control unit may be provided with a predetermined feature that enables a first avatar corresponding to a first user using the first user terminal to be distinguished from a first avatar corresponding to a first user not using the first user terminal in the virtual space.

[0062] (5) One aspect of the present embodiment is an avatar control system as described in any one of (1) to (4), which may further include an avatar generation unit that generates a corresponding avatar based on a captured image of a user as the first user or the second user.

[0063] (6) One aspect of this embodiment is the avatar control system described in (5), wherein the avatar generation unit may generate a corresponding avatar based on an image obtained by capturing an image of the first user when the first user enters the first real space.

[0064] (7) One aspect of the present embodiment is an avatar control method in an avatar control system, the avatar control method including: a virtual space providing step in which a virtual space providing unit outputs a virtual space simulating a first real space of a predetermined range on a user terminal; and an avatar control step in which an avatar control unit identifies a real position in the first real space of a first user detected by a sensor that detects a first user existing as someone belonging to the first real space, causes a first avatar corresponding to the first user to exist at a position in the virtual space corresponding to the real position, and controls communication between the first avatar and a second avatar caused to exist in the virtual space corresponding to a second user existing in a second real space other than the first real space.

[0065] (8) One aspect of this embodiment is a program for causing a computer in an avatar control system to function as a virtual space providing unit that outputs a virtual space simulating a predetermined range of a first real space on a user terminal, an avatar control unit that identifies the real position in the first real space of a first user detected by a sensor that detects a first user existing as someone belonging to the first real space, causes a first avatar corresponding to the first user to exist at a position in the virtual space corresponding to the real position, and controls communication between the first avatar and a second avatar that exists in the virtual space corresponding to a second user existing in a second real space other than the first real space. [Explanation of symbols]

[0066] 10 camera, 20 gateway, 100 smart glasses, 200 virtual space management server, 201 communication unit, 202 control unit, 203 storage unit, 221 avatar generation unit, 222 virtual space provision unit, 223 avatar control unit, 231 store clerk user information storage unit, 232 customer user information storage unit, 233 captured image position information storage unit, 234 virtual space data storage unit, 300 user terminal

Claims

1. a virtual space providing unit that outputs a virtual space simulating a predetermined range of a first real space on a user terminal; A sensor for detecting a first user who exists as a person belonging to the first real space; an avatar control unit that specifies a real position in the first real space of a first user detected by the sensor, causes a first avatar corresponding to the first user to exist at a position in the virtual space corresponding to the real position, and controls communication between the first avatar and a second avatar that exists in the virtual space corresponding to a second user who exists in a second real space other than the first real space; An avatar control system comprising:

2. the avatar control unit controls a corresponding first avatar to act in the virtual space in response to a predetermined operation performed by the first user on a first user terminal in the first real space; In response to a predetermined operation performed on a second user terminal by the second user in the second real space, a corresponding second avatar is controlled to act in the virtual space. The avatar control system of claim 1 .

3. The first user terminal is a wearable terminal that can be worn by the first user. The avatar control system of claim 2 .

4. The avatar control unit assigns a predetermined aspect that enables a first avatar corresponding to a first user who uses the first user terminal and a first avatar corresponding to a first user who does not use the first user terminal to be distinguished from each other in the virtual space.

4. An avatar control system according to claim 2 or 3.

5. The image capturing apparatus further includes an avatar generating unit configured to generate a corresponding avatar based on a captured image of the first user or the second user.

4. An avatar control system according to any one of claims 1 to 3.

6. The avatar generation unit generates a corresponding avatar based on a captured image obtained by capturing an image of the first user when the first user enters the first real space. The avatar control system of claim 5.

7. An avatar control method in an avatar control system, comprising: a virtual space providing step in which a virtual space providing unit outputs a virtual space simulating a predetermined range of a first real space on a user terminal; an avatar control step in which an avatar control unit specifies a real position in the first real space of a first user detected by a sensor that detects a first user existing as a person belonging to the first real space, causes a first avatar corresponding to the first user to exist in a position in the virtual space corresponding to the real position, and controls communication between the first avatar and a second avatar that exists in the virtual space corresponding to a second user existing in a second real space other than the first real space; 23. An avatar control method comprising:

8. A computer in an avatar control system, a virtual space providing unit that outputs a virtual space simulating a predetermined range of the first real space on the user terminal; an avatar control unit that specifies a real position in the first real space of a first user detected by a sensor that detects a first user existing as a person belonging to the first real space, causes a first avatar corresponding to the first user to exist in a position in the virtual space that corresponds to the real position, and controls communication between the first avatar and a second avatar that exists in the virtual space corresponding to a second user existing in a second real space other than the first real space. A program to function as a

Images