program

Abstract

To provide a program by which information on corresponding situation of a real space can be provided through a virtual space.SOLUTION: A program causes a computer to function as: virtual space defining means for defining a virtual space corresponding to a real space; image generation means for generating a virtual space image of the virtual space captured by a virtual camera; image display means for displaying the virtual space image on a first user terminal; and arranging means for arranging a user character operated through a second user terminal, which is different from the first user terminal, in the virtual space. The image generation means switches display mode of the user character to be included in the virtual space image, every time a selection operation to select one of operation-enabled state in which the user character can be operated and operation-disabled state in which the user character cannot be operated is performed on the second user terminal.SELECTED DRAWING: Figure 19

Description

[Technical field]

[0001] The present invention relates to a program. [Background technology]

[0002] So-called VR technology has been known for some time, which provides content corresponding to a specific location in the real world (e.g., a town, a commercial facility, a shop, an event venue, a theme park, a park, etc.) to a user who is not in that location (hereinafter referred to as "real space") (e.g., Patent Document 1).

[0003] More specifically, an image cut out from a virtual space corresponding to the real space is displayed on the user terminal, and the viewpoint of the displayed image is moved based on a movement operation on the user terminal. This allows the user operating the user terminal to have an experience as if they were in the real space, without actually going there. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2022-122810 Summary of the Invention [Problem to be solved by the invention]

[0005] However, it is difficult to know whether other user characters are operable or not through the virtual space. This causes stress for users when they send messages to other user characters (when they talk to other user characters) and do not receive a reply.

[0006] The present invention has been made in view of the above circumstances, and has an object to provide a program capable of notifying a corresponding real-space situation through a virtual space. [Means for solving the problem]

[0007] In order to solve the above problems, the program according to the present invention comprises: First user terminal A virtual space definition means for defining a virtual space corresponding to a real space, an image generation means for generating a virtual space image by capturing an image of the virtual space with a virtual camera, and a display means for displaying the virtual space image. The above an image display means for displaying an image on a first user terminal; and an arrangement means for arranging a user character, which is operated through a second user terminal different from the first user terminal, in the virtual space. a control means for controlling input of a message to be output from the second user terminal; and the image generating means switches the display mode of the user character included in the virtual space image to a display mode indicating the selected state every time the second user terminal receives from the user a selection operation of selecting either an operable state in which the user character is operable or an inoperable state in which the user character is inoperable, from the user. When the user character is in the operable state, the control means permits input of the message and transmits the inputted message to the second user terminal, and when the user character is in the inoperable state, rejects input of the message. . Effect of the Invention

[0008] According to the present invention, a situation in a corresponding real space can be notified through a virtual space. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram showing an overview of a system according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a diagram illustrating a hardware configuration of a server. [Diagram 3] FIG. 2 is a hardware configuration diagram of an HMD set, which is an example of a user terminal. [Figure 4] FIG. 11 is a hardware configuration diagram of a tablet terminal, which is another example of a user terminal. [Diagram 5] FIG. 1 is a diagram conceptually illustrating one aspect of a virtual space. [Figure 6] 1 is a diagram showing a YZ cross section of a field of view in a virtual space as viewed from an X direction. [Figure 7] 1 is a diagram showing an XZ cross section of a field of view in a virtual space as viewed from the Y direction. [Figure 8]FIG. 2 is a functional block diagram of a server and a user terminal. [Figure 9] 13 is a flowchart showing a process of a server according to the first embodiment. [Figure 10] 11 is a flowchart showing a process of the first user terminal according to the first embodiment. [Figure 11] 13 is a flowchart of a virtual space image generation and display process. [Figure 12] FIG. 11 is a conceptual diagram of a virtual space corresponding to steps S21 and S22. [Figure 13] FIG. 13 is a diagram showing a conceptual diagram of a virtual space and a virtual space image corresponding to step S47. [Figure 14] FIG. 13 is a diagram showing a conceptual diagram of a virtual space and a virtual space image corresponding to step S48. [Figure 15] FIG. 13 is a diagram showing a conceptual diagram of a virtual space and a virtual space image corresponding to step S45. [Figure 16] FIG. 13 is a diagram showing a conceptual diagram of a virtual space and a virtual space image corresponding to the case where No is determined in step S43. [Figure 17] 13 is an example of a message table that holds person attributes and messages in association with each other. [Figure 18] 13 is a flowchart showing a process of a second user terminal according to the second embodiment. [Figure 19] FIG. 13 illustrates a status setting screen. [Figure 20] 13 is a flowchart showing a process of a first user terminal according to the second embodiment. [Figure 21] FIG. 13 is a diagram showing a virtual space image when another person's avatar in an operable state is selected. [Figure 22] 13A and 13B are diagrams showing virtual space images when another person's avatar and a mob character that are inoperable are selected. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The system 1 according to the embodiment will be described below with reference to the drawings. Note that the embodiment of the present invention described below is an example of how the present invention is embodied, and the scope of the present invention is not limited to the scope of the embodiment. Therefore, the present invention can be implemented by making various changes to the embodiment.

[0011] [System 1 Overview] FIG. 1 is a diagram showing an overview of a system 1 according to this embodiment. As shown in FIG. 1, the system 1 mainly includes a server 10 and user terminals 20A, 20B, and 20C (hereinafter, these may be collectively referred to as "user terminals 20"). Although three user terminals 20 are illustrated in FIG. 1, examples of the user terminals 20 included in the system 1 are not limited to these. The server 10 and the user terminals 20 are connected to each other so as to be able to communicate with each other via a communication network 2. A specific example of the communication network 2 is not particularly limited, and may be, for example, the Internet, a mobile communication system (e.g., 4G, 5G, etc.), a wireless network such as Wi-Fi (registered trademark), or a combination of these.

[0012] The system 1 according to this embodiment is a system for allowing each user of a plurality of user terminals 20A to 20C to experience a common virtual space provided by the server 10. More specifically, the user terminal 20 places a user character (hereinafter, sometimes referred to as an "avatar") associated with the user in the virtual space, and causes the user terminal 20 to display an image of the virtual space viewed from a preset viewpoint.

[0013] Furthermore, the user terminal 20 operates the avatar in the virtual space in conjunction with the user's operation on the operating device. The avatar's movements include, for example, moving in the virtual space, moving various parts of the body, changing posture, changing facial expressions, speaking, and moving objects arranged in the virtual space.

[0014] Furthermore, the user terminal 20 transmits avatar data indicating the avatar's movements and status changes to the server 10 through the communication network 2. The server 10 transmits the avatar data received from the user terminal 20 to other user terminals 20 through the communication network 2. Then, the user terminal 20 updates the movements and status of the corresponding avatar in the virtual space based on the avatar data received from the server 10.

[0015] Moreover, the system 1 according to the present embodiment provides a virtual space corresponding to the real space. That is, the virtual space according to the present embodiment is a space that imitates the real space. However, the virtual space provided by the system 1 does not need to completely match the real space, and may be appropriately deformed. The real space refers to, for example, a city, a commercial facility, a shop, a theme park, an event venue, a park, a station, and the like that actually exists.

[0016] Furthermore, the system 1 (server 10) according to this embodiment is configured to be able to communicate with a detection device 3 installed in real space through the communication network 2. The detection device 3 is a device that detects people in real space. The detection device 3 then transmits detection data indicating the detection result to the server 10 through the communication network 2. Note that the type of detection device 3 connected to the system 1 is not limited to one type, and multiple types may be combined.

[0017] As an example, the detection device 3 may be a camera (fixed camera) that captures an image of the real space from above. In this case, the detection device 3 transmits the bird's-eye view image of the real space to the server 10 as detection data. As another example, the detection device 3 may be a camera (e.g., a device that measures surface temperature) that captures the face of a person entering a commercial facility, a store, a theme park, or an event venue. In this case, the detection device 3 transmits the image of the person's face as detection data to the server 10. As yet another example, the detection device 3 may be a ticket gate for entering a theme park, an event venue, or a station. In this case, the detection device 3 transmits purchase information of the ticket inserted into the ticket gate (e.g., distinction between adult or child, type of ticket, purchase amount, etc.) and the time of insertion into the ticket gate to the server 10 as detection data. However, specific examples of the detection data are not limited to these.

[0018] Then, the server 10 according to this embodiment analyzes the detection data received from the detection device 3 to identify the number, positions, and attributes of people present in the real space. Examples of the attributes of people include gender, age (adult or child), and posture (e.g., standing, walking, running). The server 10 then places non-user characters (hereinafter, referred to as "mob characters") corresponding to the identified people in the virtual space. More specifically, the server 10 transmits mob character data indicating the shape and position of the mob character to the user terminal 20 via the communication network 2. The user terminal 20 then places the mob character in the virtual section based on the mob character data received from the server 10. As a result, the mob character is displayed on the user terminal 20.

[0019] As described above, avatars and mob characters are placed in the virtual space according to this embodiment. The avatars and mob characters have in common the fact that they are human-shaped. The appearance of the avatar (for example, face shape, hairstyle, clothes, etc.) is determined by the user of the user terminal 20. The avatar operates in the virtual space in conjunction with the user's operation through the user terminal 20. In contrast, the appearance of the mob character is determined by the server 10. As an example, the server 10 may select a mob character that matches the attributes detected by the detection device 3 from among pre-registered mob character candidates. As another example, the server 10 may select a mob character that matches the attributes detected by the detection device 3 from the log of avatars that have been in the virtual space in the past. The mob character may operate in a manner determined by the server 10, or may be stopped.

[0020] [Server 10 configuration] Fig. 2 is a hardware configuration diagram of the server 10. The server 10 is realized by, for example, a general-purpose computer such as a workstation or a personal computer. As shown in Fig. 2, the server 10 mainly includes a processor 11, a memory 12, a storage 13, an input / output interface 14, and a communication interface 15. Each component of the server 10 is connected to a communication bus 19.

[0021] The processor 11 realizes processes described below by executing a series of instructions included in a server program 13P stored in the memory 12 or the storage 13. The processor 11 is realized as, for example, a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor unit (MPU), a field-programmable gate array (FPGA), or other device.

[0022] The memory 12 temporarily holds a server program 13P and data. The server program 13P is loaded from, for example, the storage 13. The data includes data input to the server 10 and data generated by the processor 11. For example, the memory 12 is realized as a RAM (Random Access Memory) or other volatile memory.

[0023] The storage 13 permanently holds the server program 13P and data. The storage 13 is realized, for example, as a ROM (Read-Only Memory), a hard disk drive, a flash memory, or other non-volatile storage device. The storage 13 may also be realized as a removable storage device such as a memory card. As yet another example, the storage 13 may be connected to the server 10 as an external storage device instead of being built into the server 10. With this configuration, for example, in a scene where a plurality of user terminals 20 are used, such as an amusement facility, it becomes possible to collectively update the server program 13P and data.

[0024] The input / output interface 14 is an interface for connecting external devices such as a monitor, an input device (e.g., a keyboard, a pointing device), an external storage device, a speaker, a camera, a microphone, a sensor, etc. to the server 10. The processor 11 communicates with the external devices through the input / output interface 14. The input / output interface 14 is realized using, for example, a Universal Serial Bus (USB), a Digital Visual Interface (DVI), a High-Definition Multimedia Interface (HDMI, registered trademark), or other terminals.

[0025] The communication interface 15 communicates with other devices (e.g., the detection device 3 and the user terminal 20) connected to the communication network 2. The communication interface 15 is realized as, for example, a wired communication interface such as a Local Area Network (LAN) or a wireless communication interface such as Wireless Fidelity (Wi-Fi), Bluetooth (registered trademark), or Near Field Communication (NFC).

[0026] [Configuration of user terminal 20] Fig. 3 is a hardware configuration diagram of an HMD (Head Mounted Display) set, which is an example of the user terminal 20. Fig. 4 is a hardware configuration diagram of a tablet terminal, which is another example of the user terminal 20. The user terminal 20 is realized as, for example, an HMD set shown in Fig. 3, a tablet terminal shown in Fig. 4, a smartphone, a feature phone, a laptop computer, a desktop computer, or the like.

[0027] As shown in Fig. 3, a user terminal 20 realized as an HMD set includes a computer 26 having a processor 21, a memory 22, a storage 23, an input / output interface 24, and a communication interface 25. The components of the computer 26 are connected to a communication bus 29. The basic configuration of the processor 21, memory 22, storage 23, input / output interface 24, communication interface 25, and communication bus 29 is the same as that of the processor 11, memory 12, storage 13, input / output interface 14, communication interface 15, and communication bus 19 shown in Fig. 2. In addition, the storage 23 holds a terminal program 23P.

[0028] The user terminal 20 realized as an HMD set also includes an HMD 30, a motion sensor 41, and an operation device 42 as external devices of the computer 26. The HMD 30, the motion sensor 41, and the operation device 42 are connected to the processor 21 via the input / output interface 24.

[0029] The HMD 30 is worn on the user's head to provide the user with a virtual space. More specifically, the HMD 30 may include both a so-called head-mounted display equipped with a monitor and a head-mounted device to which a smartphone or other terminal having a monitor can be attached. The HMD 30 mainly includes a monitor 31 (display device), a gaze sensor 32, cameras 33 and 34, a microphone 35, and a speaker 36.

[0030] As an example, the monitor 31 is realized as a non-transmissive display device. The monitor 31 is disposed on the main body of the HMD 30 so as to be located in front of both eyes of the user. The non-transmissive monitor 31 is realized as, for example, a liquid crystal monitor or an organic EL (Electro Luminescence) monitor.

[0031] As another example, the monitor 31 is realized as a transmissive display device. In this case, the HMD 30 is not a closed type that covers the user's eyes, but an open type such as glasses. The monitor 31 may include a configuration that simultaneously displays a part of an image that constitutes a virtual space and a real space. As an example, the transmissive monitor 31 may display an image of the real space captured by a camera mounted on the HMD 30. As another example, the transmissive monitor 31 may be configured to have an adjustable transmittance. The transmissive monitor 31 may set the transmittance of a part of the display area high so that the real space can be directly viewed.

[0032] Furthermore, the monitor 31 may employ the following configuration to allow the user to view a three-dimensional image. As an example, the monitor 31 may include a sub-monitor for displaying an image for the right eye and a sub-monitor for displaying an image for the left eye. As another example, the monitor 31 may be configured to display an image for the right eye and an image for the left eye as a single unit. In this case, the monitor 31 includes a high-speed shutter. The high-speed shutter operates to alternately display an image for the right eye and an image for the left eye so that the image is recognized by only one of the eyes.

[0033] The gaze sensor 32 detects the direction in which the gaze of the user's right eye and left eye is directed. That is, the gaze sensor 32 detects the gaze of the user. The gaze sensor 32 is realized by, for example, a sensor having an eye tracking function. The gaze sensor 32 preferably includes a sensor for the right eye and a sensor for the left eye. The gaze sensor 32 irradiates the user's right eye and left eye with infrared light and detects the rotation angle of each eyeball by receiving reflected light from the cornea and iris of the irradiated light. Then, the gaze sensor 32 identifies the gaze of the user based on each detected rotation angle.

[0034] The camera 33 captures an image of the upper part of the face of the user wearing the HMD 30 (more specifically, the user's eyes, eyebrows, etc.). The camera 34 captures an image of the lower part of the face of the user wearing the HMD 30 (more specifically, the user's nose, mouth, etc.). For example, the camera 33 is attached to the side of the housing of the HMD 30 that faces the user, and the camera 34 is attached to the side opposite to the side facing the user. Note that the HMD 30 may include one camera that captures an image of the entire face of the user, instead of the two cameras 33 and 34.

[0035] The microphone 35 converts the user's speech into an audio signal (electrical signal) and outputs it to the computer 26. The speaker 36 converts the audio signal output from the computer 26 into sound and outputs it to the user. Note that the HMD 30 may include earphones instead of the speaker 36.

[0036] The motion sensor 41 has a position tracking function for detecting the motion of the HMD 30. As one example, the motion sensor 41 may detect the position and inclination of the HMD 30 in real space by reading a plurality of infrared rays emitted by the HMD 30. As another example, the motion sensor 41 may be realized by a camera. In this case, the motion sensor 41 analyzes image information of the HMD 30 output from the camera to detect the position and inclination of the HMD 30. As yet another example, the motion sensor 41 may be realized by an angular velocity sensor, a geomagnetic sensor, or an acceleration sensor.

[0037] The operation device 42 is connected to the computer 26 by wire or wirelessly. The operation device 42 accepts input (operation) of commands to the computer 26 by the user. As one example, the operation device 42 may be a so-called controller that is operated while being held by the user. As another example, the operation device 42 may be configured to be wearable on the user's body or part of his / her clothing, and may detect the user's movements by a motion sensor. However, specific examples of the operation device 42 are not limited to these, and may be a keyboard, a pointing device, a touch panel, or the like.

[0038] 4, the user terminal 20 realized as a tablet terminal mainly includes a processor 21, a memory 22, a storage 23, a communication interface 25, a monitor 31, cameras 33 and 34, a microphone 35, a speaker 36, a motion sensor 41, and an operation device 42. The components of the tablet terminal are connected to a communication bus 29. The basic configuration of the processor 21, memory 22, storage 23, communication interface 25, monitor 31, cameras 33 and 34, microphone 35, speaker 36, motion sensor 41, and operation device 42 is the same as that of an HMD set, so the configuration specific to the tablet terminal will be described below.

[0039] The monitor 31 is provided on the surface of a flat housing. The camera 33 is a so-called in-camera that is attached to the surface of the flat housing and captures an image of the face of a user viewing the monitor 31. The camera 34 is a so-called out-camera that is attached to the back surface of the flat housing (the surface opposite to the monitor 31) and captures an image of the surroundings. The motion sensor 41 detects the motion of the housing (for example, rotation around three axes perpendicular to each other). An example of an operation device 42 suitable for a tablet terminal is a touch panel that is superimposed on the monitor 31 and receives various touch operations by the user (for example, tapping, sliding, flicking, pinching in, pinching out, etc.).

[0040] [Overview of Virtual Space 90] Fig. 5 is a diagram conceptually illustrating one aspect of the virtual space 90. Fig. 6 is a diagram illustrating a YZ cross section of the field of view 94 in the virtual space 90 as viewed from the X direction. Fig. 7 is a diagram illustrating an XZ cross section of the field of view 94 in the virtual space 90 as viewed from the Y direction.

[0041] As shown in Fig. 5, the virtual space 90 has a celestial sphere-like structure covering the entire 360-degree direction from the center C. In Fig. 5, in order to avoid complicating the explanation, the celestial sphere in the upper half of the virtual space 90 is illustrated as an example. Meshes are defined in the virtual space 90. The position of each mesh is defined in advance as a coordinate value in an XYZ coordinate system, which is a global coordinate system defined in the virtual space 90. Each partial image constituting a panoramic image 91 (still image, video, etc.) that can be deployed in the virtual space 90 is associated with a corresponding mesh in the virtual space 90.

[0042] For example, in the virtual space 90, an XYZ coordinate system is defined with the center C as the origin. The XYZ coordinate system is, for example, parallel to the real coordinate system. The horizontal direction, vertical direction (up-down direction), and front-back direction in the XYZ coordinate system are defined as the X-axis, Y-axis, and Z-axis, respectively. Therefore, the X-axis (horizontal direction) of the XYZ coordinate system is parallel to the x-axis of the real coordinate system, the Y-axis (vertical direction) of the XYZ coordinate system is parallel to the y-axis of the real coordinate system, and the Z-axis (front-back direction) of the XYZ coordinate system is parallel to the z-axis of the real coordinate system.

[0043] A virtual camera 92 associated with the user terminal 20 is placed in the virtual space 90. The position of the virtual camera 92 in the virtual space 90 corresponds to the user's viewpoint in the virtual space 90. The orientation of the virtual camera 92 corresponds to the user's line of sight (reference line of sight 93) in the virtual space 90. The processor 21 then determines a viewing area 94 (angle of view of the virtual camera 92) in the virtual space 90 based on the position and orientation of the virtual camera 92.

[0044] As shown in Fig. 6, the field of view region 94 includes an area 95 in the YZ cross section. The area 95 is a range of a polar angle α centered on the reference line of sight 93 in a vertical cross section (YZ cross section) including the reference line of sight 93 in the virtual space 90. As shown in Fig. 7, the field of view region 94 includes an area 96 in the XZ cross section. The area 96 is a range of an azimuth angle β centered on the reference line of sight 93 in a horizontal cross section (XZ cross section) including the reference line of sight 93 in the virtual space 90.

[0045] Processor 21 generates (extracts) a partial image included in field of view 94 of panoramic image 91 deployed in virtual space 90 as virtual space image 97 captured by virtual camera 92. Then, processor 21 displays the generated virtual space image 97 on monitor 31. That is, field of view 94 corresponds to the user's field of view in virtual space 90. Furthermore, field of view 94 moves in response to changes in the position and orientation of virtual camera 92 in virtual space 90, and virtual space image 97 displayed on monitor 31 is updated. That is, the user's field of view moves.

[0046] For example, the processor 21 moves the virtual camera 92 in the virtual space 90 in conjunction with a user's operation received by the operation device 42. The processor 21 also changes the orientation of the virtual camera 92 (i.e., the reference line of sight 93) in conjunction with the movement of the user terminal 20 detected by the motion sensor 41 (e.g., rotation around three mutually perpendicular axes). Furthermore, the processor 21 causes the monitor 31 to display a virtual space image 97 captured by the virtual camera 92 after the position and orientation have been changed.

[0047] [Functional block diagram of server 10 and user terminal 20] Fig. 8 is a functional block diagram of the server 10 and the user terminal 20. As shown in Fig. 8, the server program 13P loaded into the memory 12 causes the server 10 to function as an avatar synchronization means 110, a person extraction means 120, and a mob character generation means 130. In addition, the terminal program 23P loaded into the memory 22 causes the user terminal 20 (computer 26) to function as a virtual space definition means 210, a placement means 220, a camera movement means 230, an image generation means 240, and an image display means 250.

[0048] The avatar synchronization means 110 transmits (relays) avatar data received from one of the user terminals 20 via the communication network 2 to the other user terminals 20. As a result, all of the user terminals 20A to 20C hold the same avatar data. As a result, the same avatar is placed at the same position in the virtual space 90 defined in each of the user terminals 20A to 20C. That is, the avatars are synchronized in all of the user terminals 20A to 20C.

[0049] The person extraction means 120 receives the detection data from the detection device 3 through the communication network 2. Next, the person extraction means 120 extracts people in the real space from the detection data received from the detection device 3. The person extraction means 120 may also extract attributes of people in the real space from the detection data received from the detection device 3. As an example, when the detection device 3 is a camera, the person extraction means 120 extracts the number and positions of people in the real space, the gender, age, posture, etc. of the people from the image data or video data using a well-known person recognition technology. As another example, when the detection device 3 is a ticket gate, the person extraction means 120 extracts the number and ages of people in the real space from the ticket inserted into the ticket gate. However, the specific method by which the person extraction means 120 extracts people and attributes is not limited to the above example, and other well-known methods can be adopted.

[0050] The mob character generation means 130 generates mob character data indicating a mob character corresponding to a person extracted by the person extraction means 120. The mob character data indicates the appearance of the mob character (for example, body shape, facial expression, hairstyle, clothing, etc.), the position of the mob character in the virtual space 90, the posture of the mob character, etc. However, the position of the person in the real space and the position of the mob character in the virtual space 90 do not have to strictly match. Then, the mob character generation means 130 transmits the generated mob character data to all user terminals 20A to 20C. More specifically, the mob character generation means 130 generates mob characters as many as the number of people extracted by the person extraction means 120. In addition, the mob character generation means 130 generates a mob character in a manner corresponding to the attributes of the person extracted by the person extraction means 120. For example, if the extracted gender is female, a female mob character is generated, if the extracted age is child, a child mob character is generated, and if the extracted posture is sitting, a sitting mob character is generated.

[0051] The virtual space definition means 210 defines a virtual space 90 corresponding to a real space. More specifically, the virtual space definition means 210 expands virtual space data indicating the virtual space 90 in the memory 22. The virtual space data indicates, for example, a panoramic image 91 and the shapes and positions of objects (for example, buildings, plants) to be placed in the virtual space 90. The virtual space data may be downloaded in advance from the server 10 and stored in the storage 23, or may be downloaded from the server 10 when the virtual space 90 is defined. The virtual space definition means 210 may also define a virtual space 90 selected from a plurality of virtual spaces 90 through the operation device 42. A specific process for defining the virtual space 90 is already known, and therefore a detailed description thereof will be omitted.

[0052] The placement means 220 places characters in the virtual space 90 defined by the virtual space definition means 210. Here, a case where the terminal program 23P is executed in the user terminal 20A will be described. The characters placed in the virtual space 90 by the placement means 220 include the avatar of the user of the user terminal 20A (first user terminal) (hereinafter referred to as the "user's avatar"), the avatars of the users of the other user terminals 20B and 20C (second user terminals) (hereinafter referred to as the "other users' avatars"), and mob characters.

[0053] First, the arrangement means 220 arranges the principal avatar in the virtual space 90 based on the avatar data pre-stored in the storage 23. Then, the arrangement means 220 transmits the avatar data of the principal avatar to the server 10 through the communication network 2. Furthermore, when the arrangement means 220 receives a user's operation instructing an action of the principal avatar through the operation device 42, the arrangement means 220 operates the principal avatar in the virtual space 90 according to the operation. Then, the arrangement means 220 updates the avatar data to indicate a state of the principal avatar after the action, and transmits the updated avatar data to the server 10 through the communication network 2.

[0054] Furthermore, the arrangement means 220 receives avatar data of the other person's avatar from the server 10 through the communication network 2. Then, the arrangement means 220 arranges the other person's avatar in the virtual space 90 based on the avatar data received from the server 10. Furthermore, the arrangement means 220 rearranges the other person's avatar every time the avatar data is updated, thereby causing the other person's avatar to operate within the virtual space 90.

[0055] Furthermore, the placement means 220 receives the mob character data generated by the mob character generation means 130 from the server 10 via the communication network 2. Then, the placement means 220 places the mob character in the virtual space 90 based on the mob character data received from the server 10. Furthermore, the placement means 220 causes the mob character to move within the virtual space 90 by re-placing the mob character every time the mob character data is updated.

[0056] The camera moving means 230 moves the virtual camera 92 in the virtual space 90 based on the movement of the user terminal 20 detected by the movement sensor 41 or an instruction to move the avatar via the operation device 42. Moving the user terminal 20 and instructing the avatar to move via the operation device 42 are examples of movement operations. In a setting in which the relative position between the virtual camera 92 and the person's avatar does not change, the camera moving means 230 moves the virtual camera 92 following the person's avatar moved by the placement means 220. The camera moving means 230 may also move the virtual camera 92 independently of the person's avatar, according to an instruction via the movement sensor 41 or the operation device 42.

[0057] The image generating means 240 generates a virtual space image 97 by capturing an image of the virtual space 90 with the virtual camera 92. More specifically, the image generating means 240 extracts an image corresponding to the field of view 94 from the panoramic image 91 as the virtual space image 97, and stores virtual space image data indicating the virtual space image 97 in the memory 22. The image generating means 240 also controls how characters are included in the virtual space image 97.

[0058] As one example, when the number of mob characters included in the viewing area 94 (i.e., the angle of view of the virtual camera 92) is equal to or greater than a threshold value, the image generating means 240 may generate the virtual space image 97 by reducing (i.e., thinning out) the number of mob characters. As another example, the image generating means 240 may include the avatars and mob characters included in the viewing area 94 in the virtual space image in different display modes. As another example, the image generating means 240 may output a message registered in advance in association with the mob characters based on a selection operation of the mob characters executed on the operation device 42. As another example, the image generating means 240 may switch whether or not to include the mob characters in the virtual space image 97 based on an operation executed on the operation device 42. As yet another example, the image generating means 240 may switch the display mode of the avatars based on an operation executed on the operation device 42.

[0059] The image display means 250 displays the virtual space image 97 on the monitor 31. More specifically, the image display means 250 expands the virtual space image data generated by the image generation means 240 in the graphic memory of the monitor 31. Then, every time a character in the virtual space 90 is updated, the image generation means 240 generates a new virtual space image 97, and the image display means 250 displays the newly generated virtual space image 97 on the monitor 31. This allows the user of the user terminal 20 to view, through the monitor 31, an image showing the field of view of the character operating in the virtual space 90.

[0060] [Example 1] The operation of the system 1 according to the first embodiment will be described with reference to Fig. 9 to Fig. 16. In the first embodiment, it is assumed that an avatar of a user of the user terminal 20A is in a virtual space 90 corresponding to a real city. The system 1 according to the first embodiment provides the user of the user terminal 20A with an experience in the virtual space 90, and notifies the user of the situation in the corresponding real space through the virtual space 90.

[0061] FIG. 9 is a flowchart showing the process of the server 10 according to the first embodiment. FIG. 10 is a flowchart showing the process of the user terminal 20A according to the first embodiment. FIG. 11 is a flowchart showing the virtual space image generation and display process. FIG. 12 is a conceptual diagram of the virtual space 90 corresponding to steps S21 and S22. FIG. 13 is a diagram showing a conceptual diagram of the virtual space 90 and a virtual space image 97 corresponding to step S47. FIG. 14 is a diagram showing a conceptual diagram of the virtual space 90 and a virtual space image 97 corresponding to step S48. FIG. 15 is a diagram showing a conceptual diagram of the virtual space 90 and a virtual space image 97 corresponding to step S45. FIG. 16 is a diagram showing a conceptual diagram of the virtual space 90 and a virtual space image 97 corresponding to the case of No in step S43. The process of FIG. 9 by the server 10 and the process of FIG. 10 by the user terminal 20A are executed in parallel.

[0062] Server 10 repeatedly executes the process shown in Fig. 9 at predetermined time intervals. First, server 10 waits to execute the processes from step S13 onwards until it receives avatar data from user terminals 20A-20C (S11) or until it receives detection data from detection device 3 (S12). Each of user terminals 20A-20C transmits avatar data irregularly. Detection device 3 transmits detection data at predetermined time intervals. That is, the process of step S13 and the processes of steps S14-S16 are executed at arbitrary timing and frequency.

[0063] When server 10 (avatar synchronization means 110) receives avatar data from one of user terminals 20A-20C via communication network 2 (S11: Yes), server 10 transmits (relays) the avatar data received via communication network 2 to all other user terminals 20A-20C (S13). As a result, all user terminals 20A-20C hold the same avatar data. In other words, the same avatar is placed in virtual space 90 of all user terminals 20A-20C.

[0064] Furthermore, when the server 10 (person extraction means 120) receives detection data from the detection device 3 via the communication network 2 (S12: Yes), it extracts people and attributes from the received detection data (S14). Next, the server 10 (mob character generation means 130) generates a mob character corresponding to the person and attributes extracted in step S14 (S15). Furthermore, the server 10 (mob character generation means 130) transmits mob character data indicating the mob character generated in step S15 to all user terminals 20A to 20C via the communication network 2 (S16).

[0065] As shown in Fig. 10, the user terminal 20A (virtual space definition means 210) defines a virtual space 90 by expanding the virtual space data stored in the storage 23 in the memory 12 (S21). Next, the user terminal 20A (placement means 220 and camera movement means 230) places the person's avatar 50 and the virtual camera 92 in the virtual space 90 defined in step S21, as shown in Fig. 12(A), and transmits the avatar data to the server 10 through the communication network 2 (S22). The initial position of the person's avatar 50 is, for example, the center C of the virtual space 90. The position of the virtual camera 92 may be, for example, the head position of the person's avatar 50 or may be behind the person's avatar 50.

[0066] Next, when user terminal 20A (arrangement means 220) receives avatar data from server 10 via communication network 2 (S23: Yes), it arranges other person avatars 51B and 51C in virtual space 90 according to the received avatar data (S24), for example, as shown in FIG. 12(B). Note that other person avatar 51B corresponds to the avatar data transmitted from user terminal 20B, and other person avatar 51C corresponds to the avatar data transmitted from user terminal 20C. Next, user terminal 20A (image generation means 240 and image display means 250) executes a virtual space image generation and display process shown in FIG. 11 in order to reflect other person avatars 51B and 51C arranged in virtual space 90 in virtual space image 97 (S29).

[0067] As shown in FIG. 11, the user terminal 20A (image generating means 240) specifies a field of view 94 (S41). Next, the user terminal 20A (image generating means 240) determines whether or not a mob character is included in the angle of view (field of view 94) of the virtual camera 92 (S42). Since no mob character is included in the state of FIG. 12(B) (S42: No), the user terminal 20A (image generating means 240) generates a virtual space image 97 including only the avatar among the avatars and the mob characters (S50). Next, the user terminal 20A (image display means 250) displays the generated virtual space image 97 on the monitor 31 (S51).

[0068] In the state of FIG. 12(B), only the other person's avatar 51B of the other person's avatars 51B and 51C is included in the angle of view of the virtual camera 92, so that a virtual space image 97 as shown in FIG. 16(B) is displayed on the monitor 31. A toggle switch 98 may be displayed on the monitor 31 that displays the virtual space image 97. The toggle switch 98 is a screen object that accepts a user's operation to instruct whether or not to include a mob character in the virtual space image 97. The operation of switching the toggle switch 98 to the ON side is an example of a display operation that instructs the generation of a virtual space image 97 that includes a mob character included in the angle of view of the virtual camera 92. The operation of switching the toggle switch 98 to the OFF side is an example of a non-display operation that instructs the generation of a virtual space image 97 that does not include a mob character included in the angle of view of the virtual camera 92.

[0069] Furthermore, when the user terminal 20A (placement means 220) receives mob character data from the server 10 via the communication network 2 (S25: Yes), it places the mob characters 52A, 52B, 52C, and 52D in the virtual space 90 in accordance with the received mob character data (S26), for example, as shown in Fig. 13(A). Next, the user terminal 20A (image generation means 240 and image display means 250) executes a virtual space image generation and display process (S29) to reflect the mob characters 52A to 52D placed in the virtual space 90 in the virtual space image 97.

[0070] When the user terminal 20A (image generating means 240) determines that the mob characters 52A and 52B are included in the angle of view of the virtual camera 92 (S42: Yes) and determines that the toggle switch 98 is on (S43: Yes), it compares the number of mob characters 52A and 52B included in the angle of view of the virtual camera 92 with a predetermined threshold value (S44). Then, when the number of mob characters 52A and 52B included in the angle of view of the virtual camera 92 is less than the threshold value (S44: No), the user terminal 20A (image generating means 240) skips the processing of step S45 and executes the processing of step S46 and subsequent steps. That is, the user terminal 20A (image generating means 240) includes all of the mob characters 52A and 52B included in the angle of view of the virtual camera 92 in the virtual space image 97.

[0071] Next, the user terminal 20A (image generating means 240) compares the distance from the other person's avatar 51B and the mob characters 52A to the virtual camera 92, which are to be included in the virtual space image 97, with a predetermined threshold distance (S46). Then, for example, as shown in FIG. 13(A), if the distance from the other person's avatar 51B and the mob characters 52A and 52B to the virtual camera 92 is equal to or greater than the threshold distance shown by the dashed line (S46: Yes), the user terminal 20A (image generating means 240) includes the other person's avatar 51B and the mob characters 52A and 52B in the virtual space image 97 in a common display mode, as shown in FIG. 13(B) (S47). On the other hand, for example, as shown in FIG. 14(A), when the distance from the other person's avatar 51B and the mob characters 52A, 52B to the virtual camera 92 is less than the threshold distance shown by the dotted line (S46: No), the user terminal 20A (image generation means 240) includes the other person's avatar 51B and the mob characters 52A, 52B in the virtual space image 97 in a different display mode (S48), as shown in FIG. 14(B).

[0072] Here, the appearances of the other person avatar 51B and the mob characters 52A and 52B are different from each other. Therefore, the "common display mode" does not mean that the appearances of the characters are completely the same. The "common display mode" may mean, for example, that information (icons) for distinguishing between the avatar and the mob characters is not added. On the other hand, the "different display mode" may mean that information (icons) for distinguishing between the avatar and the mob characters is added. For example, in the example of FIG. 14(B), the user name "USER-B" set by the user of the user terminal 20B is added above the head of the other person avatar 51B, while the character string "MOB" is added above the heads of the mob characters 52A and 52B. On the other hand, in FIG. 13(B), the character strings above the heads of the other person avatar 51B and the mob characters 52A and 52B are omitted. However, specific examples of the common display mode and the different display mode are not limited to the above examples.

[0073] Also, as shown in FIG. 15(A), when the number of mob characters 52A, 52B, 52E, 52F, 52G, 52H, 52I, and 52J included in the angle of view of the virtual camera 92 is equal to or greater than a threshold (S44: Yes), the user terminal 20A (image generating means 240) reduces the number of mob characters 52A, 52B, 52E to 52J included in the virtual space image 97 (S45). For example, when the number of 52A, 52B, 52E to 52J is equal to or greater than a first threshold (e.g., 5) and less than a second threshold (e.g., 10), the user terminal 20A (image generating means 240) may include the first threshold of mob characters in the virtual space image 97. Also, when the number of mob characters is equal to or greater than a second threshold, the user terminal 20A (image generating means 240) may include the second threshold of mob characters in the virtual space image 97. Note that the first threshold is less than the second threshold. However, the method of reducing the number of mob characters is not limited to the above example. Also, the user terminal 20A (image generating means 240) may, for example, reduce the number of mob characters by giving priority to an area with a high density of mob characters. However, which mob characters to reduce is not limited to the above example.

[0074] Next, the user terminal 20A (image generating means 240) executes the processes of steps S46 to S48 for the other person avatar 51B and the mob characters 52A, 52E, 52G, 52H, and 52J that were not reduced in step S45. That is, the user terminal 20A (image generating means 240) adds the character string "MOB" above the heads of the mob characters 52H and 52J whose distance from the virtual camera 92 is less than the threshold (S47), and does not add the character string above the heads of the other person avatar 51B and the mob characters 52A, 52E, and 52G whose distance from the virtual camera 92 is equal to or greater than the threshold (S48).

[0075] Next, the user terminal 20A (image generation means 240) generates a virtual space image 97 including the other person avatar 51B and the mob character 52 in the example of Fig. 13(A), Fig. 14(A), or Fig. 14(A) (S49). Next, the user terminal 20A (image display means 250) displays the generated virtual space image 97 on the monitor 31 (S51), as shown in Fig. 13(B), Fig. 14(B), or Fig. 15(B).

[0076] On the other hand, when the user terminal 20A (image generating means 240) determines that the toggle switch 98 is on the OFF side (S43: No), as shown in FIG. 16(A), it executes the process of step S50 without executing the processes of steps S44 to S49. That is, the user terminal 20A (image generating means 240) generates a virtual space image 97 that includes the other person's avatar 51B included in the angle of view of the virtual camera 92, but does not include the mob characters 52A, 52B, 52E to 52J included in the angle of view of the virtual camera 92. Next, the user terminal 20A (image display means 250) displays the generated virtual space image 97 on the monitor 31 (S51), as shown in FIG. 16(B).

[0077] Returning to FIG. 10, when the user terminal 20A (the arrangement means 220 and the camera moving means 230) receives an operation instructing the operation of the principal avatar 50 through the operation device 42 (S27: Yes), the user terminal 20A moves the principal avatar 50 according to the user's operation on the operation device 42, moves the virtual camera 92 following the principal avatar 50, and transmits avatar data showing the principal avatar 50 after the operation to the server 10 through the communication network 2 (S28). In addition, the user terminal 20A (the camera moving means 230) moves the virtual camera 92 following the movement of the user terminal 20A detected by the motion sensor 41. Then, the user terminal 20A (the image generating means 240 and the image display means 250) executes a virtual space image generating & display process to generate a virtual space image 97 corresponding to the angle of view after the movement of the virtual camera 92 (S29).

[0078] The user terminal 20A repeatedly executes the processes of steps S23 to S29 until an operation instructing to end the terminal program 23P (e.g., logout) is received through the operation device 42 (S30: No). The user terminal 20A irregularly receives avatar data and mob character data, and irregularly receives user operations. That is, the processes of steps S24, S26, and S28 are executed at any timing and frequency. Then, when the user terminal 20A receives an operation instructing to end the terminal program 23P through the operation device 42 (S30: Yes), it ends the process of FIG. 10.

[0079] [Effects of Example 1] According to the first embodiment, by arranging the mob characters 52 corresponding to people in the real space in the virtual space 90, the situation in the real space (for example, the degree of congestion of people out and about in the city) can be notified to the user of the user terminal 20A who operates the user's avatar 50 in the virtual space 90. As a result, the user who is having a simulated experience in the virtual space 90 can be motivated to actually go to the real space.

[0080] Moreover, according to the first embodiment, the situation in the real space can be reported in more detail by arranging the mob characters 52 reflecting the attributes of people in the real space in the virtual space 90. As a result, the motivation of the user having a simulated experience in the virtual space 90 to actually go to the real space is further increased.

[0081] Furthermore, according to the first embodiment, the display modes of the other person's avatar 51B and the mob characters 52A and 52B whose distance from the virtual camera 92 is less than a threshold value are made different (FIG. 14), so that the user of the user terminal 20A can distinguish between the other person's avatar 51B and the mob characters 52A and 52B. This makes it possible to provide an appropriate virtual space image 97 to both a user who wants to know the situation in the real space and a user who wants to spend time in the virtual space 90 (in other words, to interact with the other person's avatar 51B).

[0082] Furthermore, according to the first embodiment, by making the display mode of the other person's avatar 51B and the mob characters 52A and 52B whose distance from the virtual camera 92 is equal to or greater than a threshold value common (FIG. 13), it is possible to prevent the virtual space image 97 from containing too much information. However, regardless of the distance from the virtual camera 92, the display modes of all other person's avatars and mob characters contained in the field of view 94 may be made different.

[0083] Furthermore, according to the first embodiment, when there are a large number of people in the real space, the number of mob characters 52 included in the virtual space image 97 is reduced, so that it is possible to prevent the number of mob characters 52 in the virtual space image 97 from becoming too large. This makes it possible to prevent the mob characters 52 from blocking the view of the user who wants to enjoy the virtual space 90.

[0084] Furthermore, according to Example 1, the display and non-display of the mob character 52 can be switched by operating the toggle switch 98, which makes it possible to more effectively prevent the mob character 52 from blocking the view of a user who wants to enjoy the virtual space 90.

[0085] [Example 2] Next, the operation of the system 1 according to the second embodiment will be described with reference to Fig. 17 to Fig. 22. Note that a detailed description of commonalities with the first embodiment will be omitted, and the description will focus on processes unique to the second embodiment. Also, the first and second embodiments can be combined in part or in whole.

[0086] In the second embodiment, it is assumed that the avatar of the user of the user terminal 20A (first user terminal) is a customer and the avatar of the user of the user terminal 20C (second user terminal) is a staff member in a virtual space 90 corresponding to a real store. The system 1 according to the second embodiment notifies the user of the user terminal 20A of the situation in the corresponding real space through the virtual space 90, and provides communication with the staff member in the real store through the virtual space 90.

[0087] FIG. 17 is an example of a message table that associates and holds person attributes and messages. FIG. 18 is a flowchart showing the processing of the user terminal 20C according to the second embodiment. FIG. 19 is a diagram showing a status setting screen. FIG. 20 is a flowchart showing the processing of the user terminal 20A according to the second embodiment. FIG. 21 is a diagram showing a virtual space image 97 when an other person's avatar 53 in an operable state is selected. FIG. 22 is a diagram showing a virtual space image 97 when an other person's avatar 53 in an inoperable state and a mob character 54 are selected.

[0088] The storage 13 of the server 10 according to the second embodiment holds a message table shown in Fig. 17. In the message table, attributes of people that can be extracted by the person extraction means 120 in step S14 are registered in association with messages. The message table is registered in advance in the server 10 by, for example, a store manager or a user of the user terminal 20C. The messages registered in the message table may be in text format or audio format.

[0089] Then, the server 10 (mob character generation means 130) extracts a message corresponding to the attributes of the generated mob character from the message table, and adds the extracted message to the mob character data (S15). Furthermore, the server 10 (mob character generation means 130) transmits the mob character data with the message added to all user terminals 20A to 20C via the communication network 2 (S16).

[0090] As shown in Fig. 18, the user terminal 20C (arrangement means 220) displays a status setting screen shown in Fig. 19 on the monitor 31 (S61). The status setting screen is a screen for setting the status of an avatar of the user of the user terminal 20C. More specifically, the status setting screen accepts an operation for selecting a value of an operable flag and setting the appearance (e.g., facial expression, gestures) and standard message of the avatar corresponding to the value of the operable flag. The operable flag can be set to a first value "ON" indicating that the avatar is in an operable state where it can be operated, or a second value "OFF" indicating that the avatar is in an inoperable state where it cannot be operated.

[0091] Then, the user terminal 20C (placement means 220) waits to execute subsequent processes until an operation for selecting the value of the operable flag, an operation for setting the appearance of the avatar, and an operation for inputting a standard message are received through the operation device 42. The operation for selecting the value of the operable flag is an example of a selection operation for selecting either an operable state or an inoperable state. Note that the interface for selecting the value of the operable flag, setting the appearance of the avatar, and receiving input of a standard message is not limited to the example of FIG. 18.

[0092] When the user terminal 20C (arrangement means 220) receives an operation to select the first value “ON” as the setting value of the operable flag through the operation device 42 (S62: Yes), it transmits avatar data including the operable flag “ON”, the appearance of the avatar set in association with the operable flag “ON”, and the fixed message input in association with the operable flag “ON” to the server 10 through the communication network 2 (S63). Also, when the user terminal 20C (arrangement means 220) receives an operation to select the second value “OFF” as the setting value of the operable flag through the operation device 42 (S64: Yes), it transmits avatar data including the operable flag “OFF”, the appearance of the avatar set in association with the operable flag “OFF”, and the fixed message input in association with the operable flag “OFF” to the server 10 through the communication network 2 (S65).

[0093] The user of the user terminal 20C can change the setting value of the operable flag at any time through the status setting screen. Then, the user terminal 20C (arrangement means 220) transmits avatar data including the operable flag to the server 10 every time the setting value of the operable flag (i.e., the operable state or inoperable state of the avatar) is changed.

[0094] The server 10 (avatar synchronization means 110) transmits the avatar data received from the user terminal 20C to all other user terminals 20A, 20B through the communication network 2 (S13). Furthermore, the user terminals 20A, 20B (placement means 220) place the other person's avatar in the virtual space 90 in accordance with the avatar data received from the server 10 (S24). Furthermore, if the other person's avatar is included in the field of view 94 (S42: Yes), the user terminals 20A, 20B (image generation means 240) generate a virtual space image 97 including the other person's avatar (S49, S50).

[0095] The process of FIG. 20 will be described below on the assumption that user terminal 20A has received mob character data including the message shown in FIG. 17 and the avatar data transmitted from user terminal 20C in steps S63 and S65.

[0096] As an example, when the operation enable flag of the avatar data transmitted from the user terminal 20C is set to a first value "ON", the user terminal 20A (image generating means 240) generates a virtual space image 97 shown in Fig. 21(A) (S49). As another example, when the operation enable flag of the avatar data transmitted from the user terminal 20C is set to a second value "OFF", the user terminal 20A (image generating means 240) generates a virtual space image 97 shown in Fig. 22(A) (however, at this point, a speech bubble including a fixed message is not displayed) (S49). Then, the user terminal 20A (image display means 250) displays the virtual space image 97 generated in step S49 on the monitor 31 (S51).

[0097] The virtual space images 97 shown in Fig. 21(A) and Fig. 22(A) are common in that they include another person's avatar 53 based on avatar data transmitted from the user terminal 20A and a mob character 54 based on mob character data transmitted from the server 10. On the other hand, the other person's avatars 53 shown in Fig. 21(A) and Fig. 22(A) differ in appearance (e.g., facial expression, gestures). Also, Fig. 21(A) and Fig. 22(A) differ in the display manner of the user name "USER-C" added to the other person's avatar 53 (grayed out in Fig. 22(A)).

[0098] That is, each time a selection operation for selecting the setting value of the operation enable flag (i.e., the operation enable state or the inoperable state of the other person's avatar 53) is executed on the user terminal 20C, the user terminal 20A (image generating means 240) switches the display mode of the other person's avatar 53. The display mode of the other person's avatar 53 that is switched according to the setting value of the operation enable flag is not limited to the examples in Fig. 21(A) and Fig. 22(A), and may be any mode that can be distinguished by the user of the user terminal 20A.

[0099] Next, when the user terminal 20A (image display means 250) receives an operation to select another person's avatar 53 shown in FIG. 21(A) or FIG. 22(A) through the operation device 42 (S71: Yes), the user terminal 20A outputs a fixed message included in the avatar data as shown in FIG. 21(B) or FIG. 22(A) (S72). In the examples of FIG. 21(B) and FIG. 22(A), a fixed message in text format is displayed in a speech bubble from the other person's avatar 53. As another example, the user terminal 20A may output a fixed message in voice format from the speaker 36. Also, in the example of FIG. 21(A), the other person's avatar 53 is selected using the pointer 55, but the specific method of selecting a character is not limited to this.

[0100] Next, the user terminal 20A (image display means 250) judges the setting value of the operable flag included in the avatar data (S73). If the user terminal 20A (image display means 250) judges that the operable flag is set to the first value "ON" (S73: Yes), the user terminal 20A displays a text box 56 for receiving message input and a [Send] button 57 on the monitor 31, as shown in Fig. 21(B). Note that the user terminal 20A is not limited to receiving a message in text format through the text box 56, and may also receive a message in voice format through the microphone 35.

[0101] Next, the user terminal 20A (arrangement means 220) waits to execute the process of step S75 until an operation of inputting a message in the text box 56 and pressing the [Send] button 57 is received through the operation device 42 (S74: No). Then, when the user terminal 20A (arrangement means 220) receives the operation of pressing the [Send] button 57 (S74: Yes), the user terminal 20A (arrangement means 220) transmits avatar data including the message inputted in the text box 56, "Please tell me what products you recommend," and the terminal ID "TERM-C" of the user terminal 20C corresponding to the other person avatar 53, to the server 10 through the communication network 2 (S75). Furthermore, when the server 10 (avatar synchronization means 110) receives the avatar data including the terminal ID, it transmits (relays) the avatar data only to the user terminal 20C identified by the terminal ID "TERM-C".

[0102] On the other hand, when the user terminal 20A (image display means 250) determines that the second value "OFF" is set to the operable flag (S73: No), the process of steps S74-S75 is skipped. That is, as shown in Fig. 22(A), the text box 56 and the [Send] button 57 are not displayed on the monitor 31, and a message cannot be sent to the user terminal 20C.

[0103] Returning to FIG. 18, when the user terminal 20C (image display means 250) receives avatar data including a message from the server 10 (S66: Yes), it outputs the message in association with the avatar of the user terminal 20A (S67). Then, the user terminal 20C (arrangement means 220) waits to execute the process of step S69 until an operation to input a reply message is received through the operation device 42 (S68: No). Next, when a reply message is input (S68: Yes), the user terminal 20C (arrangement means 220) transmits the reply message and avatar data including the terminal ID "TERM-A" of the user terminal 20A to the server 10 through the communication network 2 (S69). The processes of steps S67-S69 are common to steps S72, S74-S75.

[0104] Then, by repeating the processing of steps S72, S74-S75 executed by user terminal 20A and the processing of steps S67-S69 executed by user terminal 20C, communication is established between the user of user terminal 20A and the user of user terminal 20C (i.e., the store staff) through virtual space 90.

[0105] Returning to FIG. 20, when the user terminal 20A (image display means 250) receives an operation to select the mob character 54 shown in FIG. 21(A) through the operation device 42 (S76: Yes), it outputs the message "This is a store that is easy to enter even with children" included in the mob character data (S77), as shown in FIG. 22(B). In the example of FIG. 22(B), the message in text format is displayed in a speech bubble coming out of the mob character 54. As another example, the user terminal 20A may output a voice message from the speaker 36.

[0106] [Effects of Example 2] According to the second embodiment, whether the other person's avatar 53 can be operated by the store staff can be notified to the user of the user terminal 20A by the display mode of the other person's avatar 53. This can reduce the stress of the user of the user terminal 20A caused by not receiving a reply even when the other person's avatar 53 is sent a message (when the other person's avatar 53 is spoken to) the other person's avatar 53. In other words, in a store, there is no need to have a staff member always stationed near the user terminal 20C to make the other person's avatar 53 always operable.

[0107] According to the second embodiment, a message registered in advance in association with an attribute can be notified to the user of the user terminal 20A through the mob character 54. This allows, for example, a message promoting a store to be registered, thereby motivating the user of the user terminal 20A to go to the real store. Note that the message associated with the mob character 54 is not limited to being stored in advance in the storage 13 of the server 10, and may be automatically created using an AI function, or may quote highly rated reviews on an online review site.

[0108] [Other embodiments] In the present invention, if the user of the user terminal 20A can estimate the degree of crowding of people in the real space based on the number of mob characters placed in the virtual space 90, there is no need to have a one-to-one correspondence between the people detected by the detection device 3 and the mob characters placed in the virtual space 90. More specifically, the number of people detected by the detection device 3 and the number of mob characters placed in the virtual space 90 do not have to be the same as long as there is a positive correlation. That is, the placement means 220 may place the number of mob characters based on the detection results of the detection device 3 (e.g., the number of people, density, etc.) in the virtual space 90. More specifically, the placement means 220 may place the mob characters corresponding to the people detected by the detection device 3 in the virtual space 90.

[0109] Moreover, the objects that the placement means 220 places in the virtual space 90 are not limited to mob characters (non-user characters) corresponding to people who enter and leave the real space. Other examples of objects may be objects corresponding to things that may enter and leave the real space, such as vehicles (cars, ships, aircraft), animals, plants, merchandise, and exhibits. Furthermore, the objects placed in the virtual space 90 are not limited to one type, and may be multiple types.

[0110] As one example, by placing objects corresponding to cars in the real space in the virtual space 90 including roads and parking lots, the user of the user terminal 20A can recognize the traffic congestion situation and the availability of parking lots through the virtual space 90. As another example, by placing objects corresponding to animals in the real space and mob characters corresponding to visitors to the zoo in the virtual space 90 including a zoo or park, the user of the user terminal 20A can recognize the places where animals can be seen well through the virtual space 90.

[0111] In addition, the program according to the present invention is not limited to a single program, but may be a collection of multiple programs. In addition, the program according to the present invention is not limited to being executed by a single device, but may be executed by multiple devices in a shared manner. Furthermore, the division of roles between the server 10 and the user terminal 20 is not limited to the above example. That is, part of the processing of the server 10 may be executed by the user terminal 20, and part of the processing of the user terminal 20 may be executed by the server 10.

[0112] Furthermore, a part or all of the means realized by the program can be realized by hardware such as an integrated circuit. Furthermore, the program may be provided by being recorded on a non-transitory recording medium that can be read by a computer. Examples of the recording medium include a hard disk, an SD card, a DVD, and a server on the Internet. [Explanation of symbols]

[0113] 1...system, 2...communication network, 3...detection device, 10...server, 11,21...processor, 12,22...memory, 13,23...storage, 13P...server program, 14,24...input / output interface, 15,25...communication interface, 19,29...communication bus, 20...user terminal, 23P...terminal program, 26...computer, 30...HMD, 31...monitor, 32...gaze sensor, 33,34...camera, 35...microphone, 36...speaker, 41...motion sensor, 42...operation device, 50...personal address Butter, 51, 53...other people's avatars, 52, 54...mob characters, 55...pointer, 56...text box, 57...[Send] button, 90...virtual space, 91...panoramic image, 92...virtual camera, 93...reference line of sight, 94...viewing area, 95, 96...area, 97...virtual space image, 98...toggle switch, 110...avatar synchronization means, 120...person extraction means, 130...mob character generation means, 210...virtual space definition means, 220...placement means, 230...camera movement means, 240...image generation means, 250...image display means

Claims

1. A first user terminal, a virtual space definition means for defining a virtual space corresponding to a real space; an image generating means for generating a virtual space image by capturing an image of the virtual space with a virtual camera; an image display means for displaying the virtual space image on the first user terminal; an arrangement means for arranging a user character, which is operated through a second user terminal different from the first user terminal, in the virtual space; functioning as a control means for controlling input of a message to be output from the second user terminal; the image generating means switches the display mode of the user character included in the virtual space image to a display mode indicating the selected state each time the second user terminal receives, from a user, a selection operation for selecting either an operable state in which the user character is operable or an inoperable state in which the user character is inoperable, The control means When the user character is in the operable state, permitting input of the message and transmitting the input message to the second user terminal; A program that rejects input of the message when the user character is in the inoperable state.

2. The program according to claim 1, A standard message corresponding to each of the operable state and the inoperable state is registered in advance for the user character, The image generating means outputs the fixed message associated with the current state of the user character through the user character.

Images