Information processing device, information processing system, control method thereof and program

Abstract

To provide a scheme for facilitating a participant user to feel such that an event where participant users gather is flourishing.SOLUTION: An information processing device manages information on participant users which can participate an event and which include human agents corresponding to respective users participating the event, and information on non-human agents not corresponding to the users. This information processing device includes: obtaining means for obtaining total information on the total number of the participant users and the non-human agents in the event, and aiming information corresponding to the aiming total number of the participant users and the non-human agents; and control means for performing a control to change the participating state of the non-human agent in such a way that the total number becomes close to the aiming total number when the total number is smaller than the aiming total number based on the total information obtained by the obtaining means.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing system, and a control method and program thereof. [Background technology]

[0002] In recent years, with the improvement of XR (Extended Reality) related technologies, events that attract and interact with people in virtual spaces (virtual spaces, VR spaces) through "avatars" operated by users have become very popular.

[0003] Furthermore, in recent years, the concept of "common ground," which seamlessly connects physical space (real space) and virtual space in real time, has been proposed and is attracting attention.

[0004] "Common Ground" is a concept that aims to create an environment in which humans, human agents, and non-human agents in physical and virtual spaces can share a common understanding with each other, thereby realizing a society in which humans and agents can coexist.

[0005] To achieve this, we are attempting to utilize a variety of sensing technologies to describe physical space and the people and objects that exist there as digital information.

[0006] If such a platform can be developed and utilized, events that attract visitors and allow them to interact with each other, which have previously been confined to virtual spaces, will likely transform into events that are interactively connected to physical spaces.

[0007] Patent Document 1 describes a method for providing a technology for appropriately adjusting the formation of viewer groups for viewing content. It describes how users can be grouped in combinations according to the intimacy between participants, allowing them to enjoy an event with like-minded people. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2022-177718 A [Patent Document 2] JP 2022-168339 A Summary of the Invention [Problem to be solved by the invention]

[0009] However, in Patent Document 1, participants are simply distributed among multiple event spaces at the start of the event, and no consideration is given to the increase or decrease in the number of participants during the event. Therefore, it is not possible to deal with "depopulation situations" within the event space.

[0010] Therefore, an object of the present invention is to provide a mechanism that makes it easier for participants to feel that an event that brings together participants is a success. [Means for solving the problem]

[0011] The present invention is an information processing device that manages information regarding participants, including human agents corresponding to users who are able to participate in an event and who are participating in the event, and non-human agents that do not correspond to the users, and is characterized by comprising an acquisition means that acquires total information regarding the total number of participants and non-human agents in the event and goal information corresponding to a target for the total number of the participants and the non-human agents, and a control means that controls to change the participation status of non-human agents so that the total number approaches the target when the total number is less than the target, based on the total information acquired by the acquisition means. Effect of the Invention

[0012] In this way, the present invention can provide a mechanism for making an event that brings together participants feel like it is a success. [Brief description of the drawings]

[0013] [Figure 1] 1 is a configuration diagram showing an example of a system configuration of the present invention. [Diagram 2] FIG. 1 is a configuration diagram showing an example of a hardware configuration of the present invention. [Diagram 3] This is a diagram showing an example of an event that attracts visitors and encourages interaction by interactively connecting physical and virtual spaces. [Figure 4] This is a diagram showing an example of the relationship between space, images, and human agents. [Diagram 5] FIG. 1 is a configuration diagram showing an example of coordination between humans and agents. [Figure 6] FIG. 13 is a configuration diagram showing an example of agent switching in a virtual venue 80. [Figure 7] This is an example of a flow assuming the implementation of FIGS. 3 to 6, 9, and 10. [Figure 8] This is an example of a flow assuming the implementation of FIGS. 3 to 6, 9, and 10. [Figure 9] FIG. 2 is a diagram showing an example of a display screen for receiving input from a user regarding login and system settings. [Figure 10] FIG. 1 is a configuration diagram showing an example of coordination between humans and agents. [Figure 11] FIG. 2 is a configuration diagram showing an example of a data table of the present invention. [Figure 12] This is a diagram showing examples of events for attracting and networking participants, both in cases with a large number of participants and in cases with a small number of participants. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0015] In the following embodiments, operating devices such as robots, drones, and mobility in physical space that perform actions according to user input actions (for example, operations related to input based on piloting, operation, and user tracking information) are referred to as "physical HA (Human Agent)". Also, objects such as avatars in VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), and the like that perform actions according to user input actions are referred to as "virtual HA". Both physical HA and virtual HA are collectively referred to as "human agent (HA)".

[0016] Furthermore, in this embodiment, a user can participate in either or both of events in a physical space and events in a virtual space through a human agent, depending on the form of the user.

[0017] Therefore, in this embodiment, the "participants" of the event include not only users (humans 61 described below and other living organisms (non-human organisms)) who are living organisms who can participate in the event held in a physical space (physical venue 60 described below), but also human agents corresponding to the above-mentioned users.

[0018] Additionally, devices such as robots, drones, and mobility that can participate in events but are not subject to actions in response to user actions are referred to as "physical NHA (Non Human Agent)." Similarly, objects such as avatars in VR, AR, and MR that can participate in events but are not subject to actions in response to user actions are referred to as "virtual NHA." Both physical NHA and virtual NHA are collectively referred to as "non-human agents (NHA)."

[0019] As described above, in this embodiment, non-human agents, like human agents, can participate in either or both of events in the physical space and events in the virtual space depending on the form of the non-human agents.

[0020] In addition, the physical HA and physical NHA are collectively referred to as the "physical agent," and the virtual HA and virtual NHA are collectively referred to as the "virtual agent."

[0021] FIG. 1 is a diagram showing an example of a system configuration of the present invention.

[0022] The information processing terminal 10 is a terminal for viewing images of physical space and "Extended Reality, Cross Reality (XR) images (including Augmented Reality (AR) images, Mixed Reality (MR) images, Virtual Reality (VR) images, etc.)" and for receiving input for operating a human agent, and is a terminal that can play the role of an image device and an input device. Note that there do not necessarily have to be multiple terminals, and a single terminal may be sufficient.

[0023] The means of viewing and input reception can utilize various types of existing terminals, and specific examples include a PC 11 (desktop PC or laptop PC), a mobile terminal 12 (smartphone or tablet terminal), a head-mounted display 13 (HMD, VR goggles, headset, etc.), a tracker 14, etc.

[0024] The tracker 14 is not limited to a device worn on the body that directly detects acceleration, angular velocity, etc., but also includes an indirect mechanism that analyzes video information and distance measurement information obtained by the sensing device 30 described below and uses this as input.

[0025] The information processing device 20 is a device that generates and controls agents and XR images (VR images, AR images, MR images, etc.). Specifically, for example, it manages user login and logout, generates and manages virtual spaces, and controls the placement and operation of various agents in images of physical spaces and virtual spaces. Note that, in the configuration diagram, it is depicted as one device, and for example, a PC or a server can be used, but as long as it is a mechanism that executes necessary and sufficient information processing, there may be multiple processing entities. Therefore, for example, the information processing entities may be divided by function, and an embodiment in which information processing is distributed, such as a DAO (distributed autonomous organization), may be adopted.

[0026] The sensing device 30 is a device that acquires various information related to video (photographed images) and features, and acquires various information for grasping the state and circumstances of a physical space, such as the shape, position, behavior, quantity, and density of objects present therein. Specifically, a method of acquiring three-dimensional data using a network camera 31 or LiDAR 32 (Light Detection And Ranging) that can acquire depth information is preferably envisaged, but any suitable means may be used as long as it is suited to the purpose, and various known suitable means corresponding to the embodiment may be used, such as a thermo camera, an infrared sensor, or a free viewpoint video system using multiple network cameras.

[0027] As mentioned above, the physical agents 40 are devices such as robots, drones, and mobility that exist in the physical space. Among these, those that operate in response to operational instructions based on the user's behavior, operation, and piloting are called "physical HAs 41," and those that operate autonomously are called "physical NHAs 42."

[0028] Although the term "device" was used above, the physical agent 40 is depicted in FIG. 1 as a bipedal robot, but this is merely an example, and includes various shapes, forms, and expressions that allow humans to identify "an agent" in physical space. For example, the agent may be in the form of a figurine or a three-dimensional image. As an example of expression, a method of presentation that appeals to sensory organs other than vision, such as placing multiple speakers at an event venue and outputting three-dimensional "footsteps" corresponding to the agent's movement and position information, to create the illusion of an invisible man, is also conceivable. (Not shown)

[0029] The physical agent 40 is provided to be movable in the physical space. Although the illustration in Fig. 1 is suitable for bipedal walking, the moving form can be selected from various forms such as multi-legged, wheeled, caterpillar, flying, etc.

[0030] The communication network 50 is a network that communicatively connects hardware required for implementing the present invention (for example, the information processing terminal 10, the information processing device 20, the sensing device 30, the physical agent 40, etc.).

[0031] Moreover, various existing technologies can be used as the communication method of the communication network 50. Furthermore, the communication network 50 may be wired or wireless, and may be a WAN (Wide Area Network) or a LAN (Local Area Network).

[0032] 2 is a block diagram showing an example of the hardware configuration of the present invention. The hardware in the present invention includes at least an information processing terminal 10, an information processing device 20, a sensing device 30, and a physical agent 40.

[0033] Reference numeral 201 denotes a CPU, which loads into a RAM 203 and executes programs stored in a program ROM of a ROM 202 or in an external memory 211, thereby controlling each device connected to a system bus 204. Font ROM of the ROM 202 stores font data and the like, and data ROM of the ROM 202 stores various data. Reference numeral 203 denotes a RAM, which functions as the main memory, work area, etc. of the CPU 201. Reference numeral 202 denotes a keyboard controller (input controller), which controls inputs from 209 and a pointing device (not shown), such as a touchpad.

[0034] The input controller 205 controls the input device 209. The input device 209 also includes a touch panel (touch screen), and in such a case, the input device 209 also serves as a display unit 210 (a display device for displaying a display screen that displays an image).

[0035] The input device 209 is not limited to the above, and may be any known device. As an example, a method is envisaged in which the motion of a target is acquired and analyzed using a sensing unit 240 described below, and the acquired motion is used as an input signal. Specifically, as described above, a method is conceivable in which biological motion data is acquired using a depth camera equipped with a depth meter, and the acquired motion is linked to the motion of a human agent, or a method is conceivable in which a predetermined gesture is detected using an acceleration sensor, and the detected motion is identified as a predetermined command, and the input signal is used to operate the terminal.

[0036] A video card (VC) 206 controls the display on a display unit 210 .

[0037] Reference numeral 207 denotes a memory controller that controls access to an external memory 211 (such as a hard disk), and 208 denotes a network interface card (NIC) that controls communication on the communication network 140.

[0038] The GPS unit controller 220 is a controller that controls the GPS unit 221 .

[0039] The GPS unit 221 is a unit that combines the mechanisms required to operate the Global Positioning System (GPS). This unit enables each piece of hardware to obtain information about the aircraft's current position and altitude on the Earth.

[0040] In this embodiment, the GPS unit controller 220 and the GPS unit 221 are provided in the PC 11, the mobile terminal 12, and the physical agent 40, but may also be provided in other information processing terminals 10, information processing devices 20, and sensing devices 30.

[0041] The camera unit controller 230 is a controller that controls the camera unit 231 .

[0042] The camera unit 231 is a unit for acquiring images, and various devices can be used depending on the purpose of the image to be captured with what equipment. Specifically, in addition to general digital cameras and digital video cameras that convert visible light into image data, infrared cameras and hyperspectral cameras can also be considered. In any case, it is sufficient to have a mechanism for capturing external information for constructing an image in some way.

[0043] The camera unit controller 230 and the camera unit 231 can be provided in the PC 11, the mobile terminal 12, the head mounted display 13, the network camera 31, or the physical agent 40, but may also be provided in other information processing terminals 10, information processing devices 20, or sensing devices 30.

[0044] The sensing unit controller 240 is a controller that controls the sensing unit 241 .

[0045] The sensing unit 241 is a unit that acquires various types of information, and in particular, the sensing device 30 is provided with a unit that acquires various types of information about features as described above. The camera unit 231 may also serve as this unit, and any unit having the necessary and sufficient functions to grasp the state and circumstances, such as the shape, position, behavior, quantity, and density, of the physical space and objects present therein, can be used.

[0046] In particular, the physical agent 40 can use a unit having necessary and sufficient functions for detecting various states and situations of the hardware and its surroundings.

[0047] Specifically, known sensors depending on the application, such as an acceleration sensor, gyroscope, magnetometer, temperature sensor, proximity sensor, barometer, brightness sensor, and ambient light sensor, may be used, and the GPS unit controller 220, GPS unit 221, camera unit controller 230, camera unit 231, etc. may combine these functions.

[0048] Additionally, the information processing terminal 10, the information processing device 20, the sensing device 30, and the physical agent 40 are each provided with hardware suitable for their respective uses.

[0049] The mobile unit controller 250 is a controller that controls the mobile unit 251 .

[0050] The mobile unit 251 is a unit that combines mechanisms required for moving hardware (walking, running, flying, underwater propulsion, etc.) based on the user's behavior, the operation / operation received from the user, or an autonomous program. In this embodiment, the mobile unit 251 is provided in the physical agent 40, but may be provided in the information processing terminal 10, the information processing device 20, or the sensing device 30.

[0051] As described above, various forms of movement are assumed for the mobile unit 251, and units of various forms are assumed to be able to move according to the environment, such as walking, running, flying, etc.

[0052] The speaker unit controller 260 is a controller that controls the speaker unit 261 .

[0053] The speaker unit 261 is a unit having a configuration necessary for inputting and outputting sound, and is preferably provided in the information processing terminal 10 and the physical agent 40, but may also be provided in other devices as necessary.

[0054] Figure 3 shows a configuration diagram of an example of an event that attracts visitors and encourages interaction by interactively connecting physical and virtual spaces.

[0055] Fig. 3(a) shows an example of a physical venue 60 in a physical space, Fig. 3(b) shows details of the equipment of a person 61a in the physical venue 60, and in the balloon connected to Fig. 13(a), an example of an AR image or an MR image (collectively expressed as "AR / MR image") in which a real-life image of the physical venue 60 and a CG are superimposed on the head-mounted display 13 being worn is diagrammatically illustrated. Fig. 3(c) shows a user operating the information processing terminal 10 and an example of a VR image of a virtual venue 80 displayed on the display unit 210.

[0056] The virtual venue 80 is reproduced in a shape and size that is roughly similar to that of the physical venue 60, and is set up as a virtual world known as a digital twin or mirror world.

[0057] In addition, the features of the physical venue 60, the AR / MR images, and the features of the virtual venue 80 (particularly the space, and the positions, postures, behaviors, etc. of people and agents) are interactively synchronized.

[0058] Specifically, information about the features in the physical venue 60 is collected by the sensing device 30, the tracker 14 worn by the person 61a, and the sensing unit 241 built into the human agent 41b, and this information is reflected in the features in the virtual space. Similarly, feature information in the virtual venue 80 and the CG of the AR / MR images is also managed by the information processing device 20 and reflected in the behavior of the agents in the physical space.

[0059] In this way, an environment has been created in which the physical and the virtual can work together interactively, so in Figure 3, the human 61a and virtual HA81a, the physical HA41b and virtual HA81b, the physical HA81c and virtual HA81c in the AR / MR image, the physical NHA42d and virtual NHE82d, and the physical NHA82e and virtual NHA82e in the AR / MR image are all set to behave in a synchronized manner.

[0060] For example, suppose that a person 61a in a physical venue 60 wears a head-mounted display 13a and a tracker 14a as shown in Fig. 6(b) and raises his / her right hand while previewing an AR / MR video. Then, information acquired by the network camera 31 or LiDER 32, or information acquired by the tracker 14 worn by the person 61a, is reflected as a motion instruction for a virtual HA 81a in the VR space synchronized with the behavior of the person 61a, and the virtual HA 81a is also controlled to interactively raise its right hand.

[0061] Similarly, for example, when the mobile terminal 12b receives an operation input to move the virtual HA81b 10 steps in the X direction by touch input from a person 61b who is viewing a VR video on the display unit 210, the synchronized physical HA41b and virtual HA81b are controlled to execute an action of moving 10 steps in their respective spaces.

[0062] Furthermore, for example, when the head mounted display 13c receives an operational input from a person 61c previewing a VR video on the display unit 210 to make the virtual HA81c raise its arms in the air, the virtual HA81c of the synchronized AR / MR video and the virtual HA81c of the VR image are controlled to perform a raise-the-arm action in their respective spaces.

[0063] As described above, Figure 3 shows AR / MR images being displayed on the display unit 210 of the head mounted display 13a, and VR images being displayed on the mobile terminal 12b and the head mounted display 13c. However, the system may be such that either the AR image / MR image or the VR image is displayed, or that the two can be switched between. As an alternative embodiment to the above, the present invention may be implemented in an event that attracts customers only in a virtual venue 80. In such an embodiment, since there is no linked physical venue 60, the creator can set the virtual venue 80 as desired, and the present invention is implemented according to the spatial constraints set for that venue. Conversely to the above embodiment, the present invention may be implemented as an event that attracts customers only at the physical venue 60. In such an embodiment, since there is no linked virtual venue 80, the present invention is implemented according to the spatial constraints set as the venue.

[0064] Next, we will explain about non-human agents (NHA).

[0065] The physical venue 60, the physical NHA42d of the AR / MR images, and the virtual NHA82d of the virtual venue 80 operate based on control instructions from the information processing device 20 and programs for executing autonomous actions of the information processing unit built into the physical NHA42d (general operations performed by the CPU201 loading and executing a control program stored in an external memory into the RAM203).

[0066] Next, agent switching will be described.

[0067] The physical HA41 and the physical NHA42 are provided so as to be switchable upon receiving a switching instruction from the information processing terminal 10, the information processing terminal 20, etc. For example, when a user who operates the physical HA41b in Fig. 3(a) logs out, the physical HA41b can switch its mode to operate as the physical NHA42b upon receiving a switching instruction from the information processing terminal 10, the information processing terminal 20, etc. Similarly, when a login of a user who wants to operate the physical NHA42d is received, the physical NHA42d can switch its mode to operate as the physical HA41d upon receiving a switching instruction from the information processing terminal 10, the information processing terminal 20, etc.

[0068] The virtual HA 81 and the virtual NHA 82 are provided so as to be switchable in response to a switching instruction from the information processing terminal 10, the information processing terminal 20, or the like.

[0069] 3(c), when a user who operates the virtual HA 81c logs out, the virtual HA 81c can switch modes to operate as the virtual NHA 82c upon receiving a switching instruction from the information processing terminal 10, the information processing terminal 20, etc. Similarly, when a login is received from a user who wants to operate the virtual NHA 82e, the virtual NHA 82e can switch modes to operate as the virtual HA 81e upon receiving a switching instruction from the information processing terminal 10, the information processing terminal 20, etc.

[0070] By configuring it in this way, it is possible to flexibly respond to an increase or decrease in the number of event participants without changing the apparent density of participants.

[0071] Figure 4 is a diagram showing an example of the relationship between space, images, and human agents.

[0072] FIG. 4 shows the correspondence between the person 61, physical HA 41, physical NHA 42, virtual HA 81, and virtual NHA 82 in FIG. 3 and the physical venue 60, AR / MR images, and virtual venue 80.

[0073] A human 61, a physical HA41, and a physical NHA42 can exist in the physical space 60. The physical HA41 and the physical NHA42 are provided so as to be switchable between each other (the dotted arrow (A) indicates that they can be switched between each other).

[0074] A virtual HA 81 and a virtual NHA 82 can exist in the virtual venue 80. The virtual HA 81 and the virtual NHA 82 are provided so as to be switchable between each other (dotted arrow (B)).

[0075] The AR / MR video is a video in which the information processing device 20 superimposes CG (Computer Graphics, objects, avatars) of virtual HA81 and virtual NHA82 onto real-life video including a person 61, physical HA41, and physical NHA42 present in a physical venue 60 captured by a network camera 31. The physical HA41 and physical NHA42 (dotted arrow (C)) and the virtual HA81 and virtual NHA82 (dotted arrow (D)) are provided so that they can be switched between each other.

[0076] Next, we will discuss synchronization between people and agents in physical and virtual spaces.

[0077] As described above, the person 61 in the physical venue 60 is photographed, sensed, and tracked by the tracker 14 and the sensing device 30. Therefore, the person 61 can appear as a live-action image in the AR / MR video, and the virtual HA 81 whose behavior is synchronized with the live-action image of the person 61 can be superimposed on the AR / MR video (solid line (e)). Furthermore, the behavior of the person 61 in the physical venue 60 is synchronized with the virtual HA 81 in the virtual venue 80 (solid line (e)). However, although it is possible to reflect the behavior of the person 61 in the behavior of the virtual HA 81, the reverse is basically difficult (although this can be achieved by having the person 61 wear a powered suit or a work support suit and feeding back the behavior of the virtual HA 81 to this device).

[0078] The physical HA41 in the physical venue 60 can appear as live-action footage in the AR / MR video, and a virtual HA81 whose behavior is synchronized with the live-action footage of the physical HA41 can be superimposed and displayed in the AR / MR video (solid line (g)). Also, the behavior of the physical HA41 in the physical venue 60 is mutually synchronized with the virtual HA81 in the virtual venue 80 (solid line (h)).

[0079] The physical NHA 42 in the physical venue 60 can appear as live-action footage in the AR / MR video, and a virtual NHA 82 whose behavior is synchronized with the live-action footage of the physical NHA 42 can be superimposed and displayed in the AR / MR video (solid line (R)). Also, the behavior of the physical NHA 42 in the physical venue 60 is mutually synchronized with the virtual NHA 82 in the virtual venue 80 (solid line (J)).

[0080] FIG. 5 is a configuration diagram showing an example of the coordination of humans and agents.

[0081] The optimum number of people 90 is the value (target value, target information) of the "optimum number (or density, seat occupancy rate (utilization rate), number of seated people, etc.)" of people or agents in the AR / MR images based on the images of the physical venue 60 or in the virtual venue 80, which is determined in advance by the event organizer (administrator) so that the event at the physical venue 60 or virtual venue 80 does not become either "underpopulated" or "overcrowded."

[0082] The ideal number and density of participants for the appropriate number of people 90 will vary greatly depending on the nature and type of event (for example, a festival, a live music concert, an academic presentation, etc.), but it is preferable to set the "number and density of participants" to a level that makes participants feel that the event is a success.

[0083] In particular, when it comes to events that attract and engage with people, regardless of the type of venue (physical or virtual), event participants expect the event to be a success, so a situation where there are few participants and the venue is deserted is not desirable, as participants will feel that the event is deserted.

[0084] Therefore, if participation does not reach the target number, it is desirable to alleviate this problem by adding non-human agents to the event.

[0085] On the other hand, an "overcrowded situation" can cause problems such as participants finding it difficult to move around the venue, being unable to see their surroundings due to obstructed visibility, and others being more likely to invade personal space, making people feel more oppressed. Also, from the organizers' perspective, an overcrowded situation is undesirable because it increases the amount of calculations required and puts a strain on the server. Therefore, if the number of participants exceeds the target value, it is desirable to resolve this problem by reducing the number of non-human agents participating in the event.

[0086] By setting things up in the above way, events for attracting and interacting with people can be balanced at an ideal number of people and density, neither too crowded nor too sparsely populated, creating an ideal environment for both participants and organizers.

[0087] In response to this, the aforementioned Patent Document 1 describes a method for providing technology that appropriately adjusts the formation of viewer groups that watch content, and describes how by grouping users in combinations based on the intimacy between participants, participants can enjoy an event with like-minded people.

[0088] Therefore, although it may be possible to "optimally alleviate the overcrowding of participants" at an event, it is not possible to realize the ideal state that the present invention aims for. In addition, it is necessary to collect and manage the intimacy between participants as data in advance in some way, and this method cannot be used when there is only one event space, so there are still issues with the high hurdles of prerequisites for operation and application.

[0089] Patent document 2 also presents a mechanism for supplementing missing characters when a player is short of characters in an in-game event that features characters (game objects) owned by the player, and describes how the shortage can be resolved by calling up the missing objects from a database that already has them.

[0090] Therefore, it may be possible to "optimally resolve the shortage of characters" during an event, but the event in Patent Document 2 is merely "simple content that the player can view alone" when certain conditions are met.

[0091] Therefore, although it can solve the problem of "lack of characters owned by players" to appear in "specific content (events)" in the game, it cannot solve the "underpopulation situation" of events to attract customers and exchange. Furthermore, there is no mention of a mechanism to deal with the "overcrowding situation" of events to attract customers and exchange, and it cannot be said that it is a mechanism that can solve the problem of the present invention.

[0092] From the above, it can be said that the present embodiment can easily solve the technical problems mentioned above.

[0093] In FIG. 5, the appropriate number of people 90 is (X=5), but the value may have a range.

[0094] For example, you can set a range for the appropriate number of people 90, such as 5≦X≦10, and add non-human agents when the number falls below the minimum value, and withdraw or delete non-human agents when the number exceeds the maximum value. Providing flexibility in this way makes the increase or decrease in the number of event participants seem more natural, which can help increase the interest of the event.

[0095] In addition, while in this embodiment the optimum number of people, 90, is a predetermined value, the value may be configured to be changeable during the event, or the value may be determined according to a predetermined program.

[0096] For example, if the load on the server (information processing device 20) managing the event becomes high for some reason, such as an increase in calculation processing, it is possible to provide a mechanism for shifting the value of 5≦X≦10 to something like 3≦X≦5, thereby reducing the load for controlling the non-human agents and thereby reducing the risk of the information processing device 20 going down. This is because it is possible to realize stable continuation of the event and contribute to increasing the interest of the event.

[0097] In this way, if a mechanism is provided that can acquire the computational load of the information processing device and the appropriate number of people 90 can be adjusted according to the increase or decrease in the load, the server can be operated more stably. [About situations α and β] Situation α indicates an example of an event being “sparsely populated.”

[0098] The AR / MR image of situation α contains three objects: one human (body) 61, one physical HA 41, and one virtual HA 81.

[0099] In addition, in the VR image of situation α, there are three virtual HAs in total: one virtual HA81 corresponding to human 61 in the AR / MR image, one virtual HA81 corresponding to physical HA41 in the AR / MR image, and one virtual HA81 corresponding to virtual HA81 in the AR / MR image.

[0100] In other words, the total number of humans, human agents, and non-human agents is (H+HA+NHA=3), which is less than the optimum number of people of 90 (X=5), and the event is in a "depopulated situation."

[0101] Situation β shows an example of a situation in which two non-human agents are deployed to alleviate the shortage of people in the “depopulated situation” of situation α.

[0102] The AR / MR image of situation β includes one human (one body) 61, one physical HA 41, and one virtual HA 81, and in addition, based on instructions from the information processing device 20, one physical NHA 42 and one virtual NHA 82 are also introduced.

[0103] Note that the "insertion" of the physical NHA 42 here refers to a state in which the physical NHA 42, which is waiting in a waiting area 62 separate from the physical venue 60 as shown in FIG. 10, moves to and enters the physical venue 60 based on instructions from the information processing device 20.

[0104] Also, the "introduction" of the virtual NHA82 here refers to the state in which the CG of the virtual NHA82 is additionally superimposed on the AR / MR image.

[0105] The VR image of the situation β indicates a state in which, in addition to the three virtual HAs 81, one virtual NHA 82 corresponding to the physical NHA 42 in the AR / MR image and one virtual NHA 82 corresponding to the virtual NHA 82 in the AR / MR image are additionally displayed in the virtual venue based on the instruction of the information processing device 20.

[0106] [Regarding situation γ and situation δ] Situation γ shows an example of an "overcrowded" event situation.

[0107] The AR / MR image of situation γ includes a total of seven objects: one human 61, two physical HA41, two virtual HA81, one physical NHA42, and one virtual NHA82.

[0108] In addition, the VR image of situation γ includes a total of five virtual HA81s corresponding to human 61, physical HA41, and virtual HA81 in the AR / MR image, respectively, and a total of two virtual NHA82s corresponding to physical NHA42 and virtual NHA82, respectively.

[0109] In other words, the optimal number of people is 90 (X=5), but the total number of humans, human agents, and non-human agents is (H+HA+NHA=7), which is greater than the optimal number of people, 90 (X=5), and the event is in an "overcrowded situation."

[0110] Situation δ shows an example of a situation in which two non-human agents are deleted and withdrawn in order to resolve the “overcrowding” of situation γ.

[0111] In the AR / MR image of situation δ, based on instructions from the information processing device 20, one physical NHA 42 has been withdrawn from the physical venue 60 (=moved to the waiting area 62) from situation γ, and one virtual NHA 82 (CG) has been deleted (deleted from display).

[0112] Therefore, the AR / MR image of situation δ consists of one human (body) 61, two physical HAs 41, and two virtual HAs 81.

[0113] In the VR video of situation δ, a total of two virtual NHAs 82 are removed, one virtual NHA 82 corresponding to the physical NHA 42 in the AR / MR video and one virtual NHA 82 corresponding to the virtual NHA 82 in the AR / MR video, based on an instruction from the information processing device 20. Therefore, in the VR video of situation δ, a total of five virtual HAs 81 exist in the virtual venue 80, corresponding respectively to the person 61, physical HA 41, and virtual HA 81 in the AR / MR video.

[0114] FIG. 6 is a configuration diagram showing an example of agent switching in the virtual venue 80. As shown in FIG.

[0115] FIG. 5 explains how to deal with "underpopulated" and "overpopulated" situations at an event, but FIG. 6 mainly explains how to switch agents in the virtual venue 80 in situations within the range of the optimum number of people 90.

[0116] FIG. 6(a) shows a virtual venue 80 with an appropriate number of people of 90 (X=3) in which three virtual HAs 81 (avatars A to C) exist.

[0117] Here, if the user operating "avatar C" in virtual HA81 logs out, the number of agents in the venue will fall below the appropriate number of 90.

[0118] So I can think of two natural ways to deal with this situation:

[0119] [First switching method] The first switching method is switching of an agent without changing an object.

[0120] As shown in FIG. 6(b), avatar C switches from virtual HA 81 to virtual NHA 82 without changing the object.

[0121] Conversely, when a user who wants to participate in the virtual venue 80 logs in in the state shown in Figure 6(b), avatar C is assigned to the logged-in user, as shown in Figure 6(a), and switches from virtual NHA 82 to virtual HA 81 without changing the appearance of the object.

[0122] The advantage of switching agents in this way without changing the object is that since there is no apparent change in the object, it is difficult for other participants to notice that a participant has entered or left the event, and this does not ruin the excitement of the event.

[0123] [Second switching method] The second switching method is agent switching accompanied by a change in object.

[0124] The first switching method cannot be implemented if the logged-in user has a license to use an avatar, but the operator does not have that right.

[0125] One possible solution is to switch between agents while changing the object, while keeping the agent's position and posture the same.

[0126] In other words, if the user controlling avatar C of virtual HA81 in Figure 6(a) logs out, the system switches from virtual HA81 to virtual NHA82 (avatar D), which has a different object, without changing the posture or coordinates of avatar C's object, as shown in Figure 6(c).

[0127] By setting it up in this way, the position and posture of the avatar do not change, so agent switching can be carried out in a natural manner while also taking licenses into consideration.

[0128] With the above switching method, event participants can enter and exit naturally in a way that makes it difficult for other participants in the vicinity to notice the increase or decrease in the number of participants, so the excitement of the event is not lost, and ultimately the excitement of the event can be increased compared to conventional technology.

[0129] 7 and 8 are examples of flows assuming the implementation of FIGS. 3 to 6 and FIGS. 9 and 10. In FIG.

[0130] FIG. 7 is a block diagram showing an example of a main flow of the present invention.

[0131] In the description of the flow of this embodiment, specifically, the CPU 201 loads a control program stored in an external memory into the RAM 203 and executes it. However, for the sake of simplicity, the description will be centered on the information processing terminal 10, the information processing device 20, the sensing device 30, and the physical agent 40.

[0132] The method of receiving an instruction from the user is not particularly limited, and various existing methods can be used. Specifically, various methods include receiving the instruction via the input device 209 (KB / button / touch panel / gesture input / input based on tracking information of user behavior obtained from inside and outside, etc.) and the display unit 210.

[0133] In step S701, the sensing device 30 acquires information about captured images and features in the physical space 60 (referring to three-dimensional data and information for grasping the position, number, density, status, etc. of the human 61 (H), physical HA41, and physical NHA42).

[0134] In step S702, the sensing device 30 transmits the information acquired in step S701 to the information processing device 20.

[0135] In step S703, the information processing device 20 receives the information transmitted by the sensing device 30 in step S702.

[0136] In step S704, the information processing device 20 generates a spatial image based on the information acquired in step S703.

[0137] The spatial image referred to here refers to an image that the user can identify as the spatial portion of the VR image or the AR / MR image. For example, in the case of the VR image, it refers to the virtual venue 80, and in the case of the AR / MR image, it is assumed that the image is a live-action image captured by the network camera 31, or a free viewpoint image based on the live-action image captured by the network camera 31. In another embodiment, the captured image and the feature image may be acquired by the display terminal 10 instead of the sensing device 30, and steps S704 and S711 described below may be performed on the display terminal 10 side, and the display image (VR image or AR / MR image) may be generated and displayed by the display terminal 10.

[0138] Furthermore, in cases where an event is held in a virtual venue 80 that is not simply a digital twin, there is no information about the physical space (physical venue 60), so in such cases steps S701 to S703 can be omitted.

[0139] In step S705, information processing terminal 10 accepts an instruction to log in to the system from the user.

[0140] In step S706, the information processing terminal 10 determines whether or not an instruction was received in step S705. If it is determined that an instruction was received, the process proceeds to step S707. On the other hand, if it is not determined that an instruction was received, the process proceeds to step S705.

[0141] In step S707, the information processing terminal 10 issues a login request to the information processing device 20.

[0142] In step S708, the information processing device 20 accepts the login request transmitted from the information processing terminal 10 in step S707 and performs login processing.

[0143] In step S709, the information processing device 20 calculates and acquires the number or density of the human 61 (H), human agents (HA: physical HA41 and virtual HA81), and non-human agents (NHA: physical NHA42 and virtual NHA82) based on the information acquired in step S703. Various existing methods can be used for the calculation and acquisition method according to the system configuration. For example, a video analysis of the captured video of the physical space 60 may be performed, or the information processing device 20 may derive the number or density from the result of the calculation processing performed to generate the VR video space 80.

[0144] In step S710, the information processing device 20 adjusts the number or density of human agents and non-human agents in the spatial image according to the number or density of humans, human agents, and non-human agents.

[0145] Details of the process will be explained in Fig. 8, but if the number is within the range of the appropriate number of people 90, a process is performed to switch or replace non-human agents with human agents in response to the login accepted in step S708. If the number is outside the range of the appropriate number of people 90, a process is performed to add human agents or non-human agents or to withdraw or delete agents in order to bring the number closer to the range.

[0146] In step S711, the information processing device 20 generates an image (display image) to be displayed on the display unit 210 of the information processing terminal 10 based on the information received in step S703, the login request received in step S708, the spatial image generated in step S704, and the adjustments performed in step S710.

[0147] The display image referred to here is a VR image or an AR / MR image to be displayed on the display unit 210 of the information processing terminal 10, and is an image in which CG such as a virtual HA81 and a virtual NHA82 are superimposed on the spatial image generated in step S704.

[0148] Furthermore, in the case of an embodiment in which a display image is generated by the information processing terminal 10, information necessary for generating the display image by the information processing terminal 10, which is information to be transmitted in the next step, such as spatial feature information, and posture and position information of people or agents, is generated in this step.

[0149] In step S712, the information processing device 20 transmits the display image generated in step S711 (in the case of an embodiment in which the display image is generated by the information processing terminal 10, information necessary for the information processing terminal 10 to generate the display image) to the information processing terminal 10.

[0150] In step S713, the information processing terminal 10 receives the information transmitted by the information processing device 20 in step S712, and displays it on the display unit 210. (In the case of an embodiment in which a display image is generated by the information processing terminal 10, a display image is further generated based on the received information and displayed on the display unit 210. Also, a part or all of the processes from step S703 to step S710, excluding step S708, may be performed by the information processing terminal 10.) In step S714, the information processing terminal 10 acquires an instruction for moving a human agent (physical HA 41 or virtual HA 81) that can be operated by the logged-in user, or the behavior of the user. Specifically, the information is acquired from the input device 209 or the sensing unit 241. Alternatively, the information may be acquired from the sensing device 30 (not shown).

[0151] In step S715, the information processing terminal 10 transmits to the information processing device 20 a request to operate the human agent based on the instruction / behavior acquired in step S714.

[0152] In step S716, the information processing device 20 accepts the request transmitted from the information processing terminal 10 in S715, and performs processing for operating the human agent.

[0153] In step S717, information processing terminal 10 accepts an instruction to log out from the system from the user.

[0154] In step S718, the information processing terminal 10 determines whether or not an instruction was received in step S717. If it is determined that an instruction was received, the process proceeds to step S719, whereas if it is determined that an instruction was not received, the process proceeds to step S713.

[0155] In step S719, the information processing terminal 10 transmits to the information processing device 20 a request to log out from the system.

[0156] In step S720, the information processing device 20 accepts the logout request transmitted from the information processing terminal 10 in step S719 and performs a logout process.

[0157] In step S721, the information processing terminal 10 judges whether or not to end the flow. If it is judged that the flow should be ended, the flow is ended, and if not, the process proceeds to step S713.

[0158] In step S722, the information processing device 20 determines whether or not to end the flow. If it is determined that the flow should be ended, the flow is ended, and if not, the process proceeds to step S703.

[0159] In step S723, the sensing device 30 determines whether or not to end the flow. If it is determined that the flow should be ended, the flow is ended, and if not, the process proceeds to step S701.

[0160] FIG. 8 is a block diagram showing an example of a subflow of the present invention.

[0161] In the explanation of this flow, for the sake of simplicity, the decision branching is explained as being triggered by the "number" of humans, human agents, and non-human agents, but this can also be interpreted as "density."

[0162] In step S801, the information processing device 20 determines whether or not there has been a change in the "number" of people, human agents, and non-human agents (total information on the total number of participants in the event (including human agents corresponding to users participating in the event) and non-human agents) (for example, whether or not a change has occurred in the parameters due to the login request acceptance in step S708, the acquisition of the number and density in step S709, or the logout acceptance in step S720). If it is determined that there has been a change, the process proceeds to step S802, and if not, the process proceeds to step S809.

[0163] In step S802, the information processing device 20 judges whether or not the total number (H+HA+NHA) of humans (H), human agents (HA), and non-human agents (NHA) is within the range of the appropriate number of people 90 (X). If it is judged to be within the range, the process proceeds to step S803, and if not, the process proceeds to step S809.

[0164] In step S803, the information processing device 20 determines whether or not to switch or replace the virtual agent. If it is determined that the virtual agent should be switched or replaced, the process proceeds to step S804, and if not, the process proceeds to step S805.

[0165] Note that the judgment in this step can be made in various ways depending on the system configuration, but as an example, the case in Fig. 9 will be described later. In this case, the login mode of the new user logging in is "logging in using a physical human agent", so the process transitions to step S805.

[0166] In addition, even if "login using a physical human agent" is selected, if there is no physical agent available, the user's information processing terminal 10 may be notified of this and the user may be asked whether or not to log in using a virtual agent. Alternatively, the display may be controlled so that "login using a physical human agent" cannot be selected when selecting the participation type selection button 904 described below.

[0167] In step S804, the information processing device 20 performs processing to switch (or replace) the virtual HA and the virtual NHA.

[0168] Taking Fig. 9 as an example, "agent replacement / switching" is set to "switch between human and non-human with the same agent", so the appearance of the already existing avatar remains the same, and a process is performed to switch virtual NHA82 to virtual HA81. Taking Fig. 6 as an example, avatar C in Fig. 6(b) will transition to avatar C in Fig. 6(a). Conversely, if the user logs out under the same conditions, taking Fig. 6 as an example, a process is performed to switch avatar C in Fig. 6(a) to avatar C in Fig. 6(b).

[0169] Also, in Fig. 9, when "agent replacement / switching" is set to "replace human and non-human with different agents," the existing avatar (virtual NHA 82) will be "replaced" by an avatar with a different appearance (virtual HA 81). Taking Fig. 6 as an example, avatar D in Fig. 6(c) will be "replaced" by avatar C in Fig. 6(a). Conversely, when the user logs out under the same conditions, taking Fig. 6 as an example, avatar C in Fig. 6(a) will be "replaced" by avatar D in Fig. 6(c).

[0170] In step S805, the information processing device 20 transmits to the physical agent 40 an instruction to switch (or swap) the physical HA and the physical NHA.

[0171] In step S806, the physical agent 40 receives the instruction to switch (or swap) the physical HA and the physical NHA transmitted by the information processing device 20 in step S805.

[0172] In step S807, the physical agent 40 judges whether or not the instruction was received in step S806. If it is judged that the instruction was received, the process proceeds to step S808. On the other hand, if it is not judged that the instruction was received, the process proceeds to step S813.

[0173] In step S808, the physical agent 40 performs a process of switching (or exchanging) the physical HA 41 and the virtual NHA 42.

[0174] Taking FIG. 9 as an example, similar to the explanation of step S804, whether to execute "switch" or "replace" is determined by whether "agent replacement / switch" selects "switch between human and non-human with the same agent" or "switch between human and non-human with different agents."

[0175] Taking Fig. 10 as an example, when a new user logs in with "switch between human and non-human with the same agent", the existing avatar's body remains the same, and the agent C (physical NHA42) in Fig. 10(b) switches to the agent C (physical HA41) in Fig. 10(a). Conversely, when the user logs out under the same conditions, the agent C (physical HA41) in Fig. 10(a) switches to the agent C (physical NHA42) in Fig. 10(b), as shown in Fig. 10(a) and Fig. 10(b).

[0176] Also, when "human and non-human are replaced by different agents", a process of physically "replacing" the agents is carried out. Therefore, as shown in Fig. 10(b), a control instruction is issued to agent D (physical NHA42) waiting in waiting area 62 (a physical space different from physical space 60) to enter the venue in physical space 60 and switch to physical HA41 (flow not shown), and a control instruction is issued to agent C (physical NHA42) to leave physical space 60 and enter waiting area 62. In other words, agent D is "replaced" by agent C, as shown in Fig. 10(b) transitioning to Fig. 10(c).

[0177] Conversely, if the user logs out under the same conditions, Figure 10(c) becomes Figure 10(b), and agent D in Figure 10(c) switches from physical HA41 to physical NHA42 and moves from physical space 60 to waiting location 62, and agent D enters physical space 60 from waiting location 62, resulting in a transition to the state of Figure 10(b), and the process of agents C and D ``switching'' is performed.

[0178] In step S809, the information processing device 20 judges whether the total number of humans, human agents, and non-human agents (H+HA+NHA) is less than the range of the appropriate number of people 90 (X). If it is judged to be less, the process proceeds to step S810, and if not, the process proceeds to step S816.

[0179] In step S810, the information processing device 20 determines whether or not to add a virtual agent. If it is determined that a virtual agent is to be added, the process proceeds to step S811, and if not, the process proceeds to step S812.

[0180] As in step S803, in the case of Fig. 9, if the login mode of the new user is "logging in using a physical human agent", the process transitions to step S812. However, if there is no physical agent that can be added, the process transitions to step S811. Also, if the login mode is "preferentially using a virtual agent", the process transitions to step S811.

[0181] Furthermore, even if the option to “log in using a physical human agent” is selected, if there is no available physical agent, the user's information processing terminal 10 may be notified of this and the user may be asked whether or not to log in using a virtual agent. Alternatively, the display may be controlled so that “log in using a physical human agent” cannot be selected when selecting the participation form selection button 904.

[0182] In step S811, the information processing device 20 performs a process of additionally inserting a virtual HA 81 or a virtual NHA.

[0183] More specifically, if the user has just logged in, a process is performed to add a virtual HA 81, and if not, a process is performed to add a virtual NHA. Taking Figure 6 as an example, if there is one agent missing from the appropriate number of agents as shown in Figure 6(d) and a new user logs in, agent C (virtual HA81) will be added to the virtual venue 80, resulting in the pattern shown in Figure 6(a); if no new user has logged in, agent C (virtual NHA82) will be added to the virtual venue 80, resulting in the pattern shown in Figure 6(b).

[0184] In step S812, the information processing device 20 transmits to the physical agent 40 an instruction to add the physical HA 41.

[0185] In step S813, the physical agent 40 receives the instruction to add the physical HA 41 transmitted by the information processing device 20 in step S812.

[0186] In step S814, the physical agent 40 judges whether or not the instruction was received in step S813. If it is judged that the instruction was received, the process proceeds to step S815. On the other hand, if it is not judged that the instruction was received, the process proceeds to step S821.

[0187] In step S815, the physical agent 40 performs a process of additionally inputting the physical HA 41 or the physical NHA 42.

[0188] Taking FIG. 10 as an example, when a new user logs in with one agent missing from the appropriate number as shown in FIG. 10(d), agent D (physical NHA42) waiting in waiting area 62 (a physical space in a different location from physical space 60) may additionally enter the venue in physical space 60 and switch to physical HA41; or when a new user has not logged in, agent D (physical NHA42) waiting in waiting area 62 may additionally enter the venue in physical space 60 as physical NHA42 to make up for the missing agent (not shown). In step S816, the information processing device 20 determines whether the number of non-human agents (NHAs) is 0. If it is determined that the number is 0, the flow ends, and if not, the flow proceeds to step S817.

[0189] In step S817, the information processing device 20 judges whether the total number of humans, human agents, and non-human agents (H+HA+NHA) is greater than the range of the appropriate number of people 90 (X). If it is judged to be greater, the process proceeds to step S818, and if not, the flow ends.

[0190] In step S818, the information processing device 20 determines whether or not to withdraw (or delete) the virtual agent. If it is determined that the virtual agent is to be withdrawn, the process proceeds to step S819, and if not, the process proceeds to step S820.

[0191] As with step S803, in the case of Figure 9, if the agent priority is "use physical agents preferentially," the process transitions to step S819, where the virtual NHA 82 is preferentially deleted. If there is no virtual NHA 82 that can be deleted, the process transitions to step S820, where the physical NHA 41 is to be withdrawn.

[0192] On the other hand, if "preferential use of virtual agents" is selected, the physical NHA 41 will be removed as a priority, and the process will transition to step S820 as a priority. If there is no physical NHA 41 that can be removed, the process will transition to step S819, in which the virtual NHA 82 will be deleted.

[0193] In step S819, the information processing device 20 performs a process of deleting the virtual NHA 82 so that the combined number (H+HA+NHA) of humans, human agents, and non-human agents approaches the range of the appropriate number of people 90 (X).

[0194] Taking FIG. 6 as an example, there are two possible patterns: one in excess of the appropriate number of people, as shown in FIG. 6(e), in which agent C (virtual NHA 82) is deleted from the virtual venue 80, and the state shown in FIG. 6(c) is assumed. Regarding the deletion process, the virtual NHA 82 may be deleted immediately, but it is preferable to have it move as if it is leaving (withdrawing) the venue, as this makes the movement look more natural.

[0195] It is not necessary to delete more than the appropriate number of people (90(X)). For example, if X=4~5, NHA=3, and H+HA+NHA=6, then it is sufficient to delete one.

[0196] In step S820, the physical agent 40 transmits a control instruction to the physical agent 40 (physical NHA 42) to withdraw so that the combined number of humans, human agents, and non-human agents (H+HA+NHA) approaches the range of the appropriate number of people 90 (X).

[0197] In step S821, the physical agent 40 receives the control instruction for withdrawal transmitted by the information processing device 20 in step S820.

[0198] In step S822, the physical agent 40 judges whether or not the instruction was received in step S821. If it is judged that the instruction was received, the process proceeds to step S823. On the other hand, if it is not judged that the instruction was received, the flow ends.

[0199] In step S823, the physical agent 40 performs a process for withdrawal so that the combined number (H+HA+NHA) of humans, human agents, and non-human agents approaches the range of the appropriate number of people 90 (X).

[0200] Taking Figure 10 as an example, in Figure 10(a), agent D (physical NHA42) waiting at waiting location 62 is in physical space 60 with agents A, B, and C (not shown), i.e., there is one more participant than the appropriate number of participants. In this case, agent D (physical NHA42) withdraws to waiting location 62, and the number of participants in physical space 60 is adjusted to the appropriate number, as shown in Figure 10(a).

[0201] Also, for example, taking the AR / MR video of FIG. 5 as an example, there is a case where the physical NHA 42 is withdrawn from the physical space (real space) in the situation γ of FIG. 5, resulting in the situation δ of FIG.

[0202] Also, there is no need to withdraw more than the appropriate number of people (X) of 90. Therefore, for example, if X=4~5, NHA=3, H+HA+NHA=6, then only 1 person should be withdrawn.

[0203] The above-described Figs. 7 and 8 are examples of flows assuming the implementation of Figs. 3 to 6 and Figs. 9 and 10. Then, based on the total information (e.g., the total number of humans and agents and the density) regarding the total number of participants (e.g., humans and human agents) and non-human agents, the event can be brought closer to an optimal state by controlling the addition, deletion, and withdrawal of non-human agents, and switching or replacing them (with human agents) (changing the participation state) so that the total number approaches the target of an optimal number of people.

[0204] 9 is a configuration diagram showing an example of a display screen for receiving input from a user regarding login and system settings. It is assumed that the UI (User Interface) is displayed mainly on the display unit 210 of the information processing terminal 10 or the information processing device 20.

[0205] FIG. 9(a) is a diagram showing an example of the login screen.

[0206] A user ID text area 901 is a text area that accepts input of an ID required for login from the user.

[0207] The password text area 902 is a text area for accepting input of a password required for login from the user.

[0208] A login execution acceptance button 903 is a button for accepting login execution from the user.

[0209] The participation form selection button 904 is an object for selectively accepting the form of participation in the event from the user. If the user wishes to participate in the venue (physical space 60) as a human (with a living body), the user can select "Log in as a human (at the venue)". If the user wishes to participate in the event via the physical HA 41, the user can select "Log in using a physical human agent". If the user wishes to participate in the event via the virtual HA 81, the user can select "Log in using a physical human agent".

[0210] Note that "(2 remaining)" indicates the number of physical agents remaining that are available as physical HA41 and have not yet been operated by the user. Therefore, for example, if there are five physical agents that can be used by switching between human agents and non-human agents, and three of them are already being used as physical HA41 by a logged-in user, "(2 remaining)" will be displayed. Therefore, a mechanism may be provided to control the display (such as making it impossible to select, or hiding the display of options) so that the user cannot select an agent when there are "0 remaining."

[0211] The display image state selection button 905 is a button for selectively accepting from the user an image to be displayed on the display unit 210. If the user wishes to display live-action image or an image in which CG is superimposed on a free viewpoint image based on the live-action image, the selection of "AR / MR image" is accepted. Also, if the user wishes to display an image in which a virtual agent is placed in a virtual space, this is an object for accepting the selection of "VR image." Note that in an embodiment in which there is no image that can be switched to display, this may not be necessary.

[0212] The priority agent selection button 906 is an object for receiving from the user a selection as to whether a physical agent or a virtual agent should be given priority for introduction into the venue.

[0213] The agent replacement switching setting button 907 is an object for accepting a selection from the user as to whether to adjust the number and density of non-human agents in response to an increase or decrease in the number of human agents as the user logs in (joining an event) and logs out (leaving an event), by switching the same agent or by replacing a different agent.

[0214] The appropriate number of people reception area 908 is an object that receives the range of 90 for the appropriate number of people from the user.

[0215] FIG. 10 is a configuration diagram showing an example of the coordination of humans and agents.

[0216] FIG. 10 shows an example of adjusting the number and density of agents in a physical space 60 with an optimal number of agents of 90 (X=3).

[0217] FIG. 10(a) shows a state in which three physical HA41 (avatars A to C) exist in a physical space 60 with an appropriate number of people of 90 (X=3).

[0218] In this case, when the agent swap setting button 907 in FIG. 9 is set to "Switch between human and non-human with the same agent," and the user controlling "avatar C" of physical HA41 logs out, a process is performed in which avatar C is switched from physical HA41 to physical NHA42 without changing the appearance of the object, as shown in FIG. 10(b).

[0219] Conversely, when a user who wants to participate in physical space 60 logs in in the state shown in Figure 10(b), avatar C is assigned to the logged-in user as shown in Figure 10(a), and a process is performed to switch from physical NHA41 to physical HA41 without changing the appearance of the object.

[0220] As with Figures 6(a) and 6(b), the advantage of not changing the appearance, as in the examples of Figures 10(a) and 10(b), is that there is no change in the appropriate number of participants of 90, there is no movement of objects, and there is no visible change, so it is difficult for other participants to tell when participants are entering or leaving, and this does not ruin the excitement of the event.

[0221] On the other hand, in the case where "Replace human and non-human with different agents" is selected for the agent replacement setting button 907 in Fig. 9, when a user who wants to participate in the physical space 60 logs in in Fig. 10(b), as indicated by the dotted arrow, a process is executed in which the physical NHA42 (avatar D) waiting in the waiting area 62 enters the physical space 60, the event venue, and is switched to the physical HA41. In exchange, the physical NHA42 (avatar C) in the physical space 60 (event venue) leaves the event venue and waits in the waiting area 62.

[0222] 10(c), when the user of avatar D logs out, as indicated by the dotted arrow, a process is executed in which physical NHA42 (avatar C) waiting in waiting area 62 enters the event venue into physical space 60. In exchange, physical HA41 (avatar D) in physical space 60 (event venue) is replaced by NHA42, leaves the event venue, and waits in waiting area 62. FIG. 11 is a diagram showing an example of a data table according to the present invention.

[0223] The data tables in Figure 11 (venue management data table 1100 and user management data table 1200) are described as an example assuming that they are managed by the information processing device 20, but they can take various forms depending on the embodiment, such as managing some or all of them by other hardware that is communicatively connected.

[0224] The venue management data table 1100 is a data table for managing information relating to venue management.

[0225] 1101 manages the number of people accepted in the appropriate number reception area 908.

[0226] 1102 manages the number of people at physical venue 60. (Obtained in S709) 1103 manages the number of physical HA41s at 60 physical venues. (Obtained in S709) 1104 manages the number of virtual HA81 for 60 physical venues. (Obtained in S709) 1105 manages the number of physical NHAs at 60 physical venues.

[0227] 1106 manages the number of physical NHA42 in waiting area 62.

[0228] 1107 manages a virtual NHA of 82 with 60 physical venues.

[0229] 1108 manages the total number of non-human agents in 60 physical venues.

[0230] 1109 manages the venue's area.

[0231] 1110 manages the density obtained by dividing the total number of people and human agents in the physical venue 60 by the area per unit (in this embodiment, the area of ​​the venue (i.e., the area within the designated area that is the venue) is 100 square meters). 1111 manages the appropriate density per a given area received in the appropriate number of people reception area 908 .

[0232] 1112 manages the difference between the appropriate number of people and the total number of humans, human agents, and non-human agents.

[0233] 1113 manages the difference between the appropriate density and the current density.

[0234] A column 1114 manages the agents to be preferentially used, which are accepted by the priority agent selection button 906 .

[0235] 1115 manages whether to switch between human agents and non-human agents (switching between humans and non-humans with the same agent) or to replace (switching between humans and non-humans with different agents) in order to approach the optimum number of people of 90, as accepted by the agent replacement setting button 907.

[0236] The user management data table 1200 is a data table for managing information relating to user management.

[0237] The device 1201 manages the user login IDs.

[0238] The numeral 1202 manages user login passwords.

[0239] A column 1203 manages the user's event participation form (accepted by pressing the participation form selection button 904).

[0240] A display image type 1204 is managed to be displayed on the information processing terminal of each user (accepted by the display image type selection button 905).

[0241] 1205 manages the position coordinates of each user's physical venue and virtual venue. 1206 manages the orientation of each user in the physical venue and the virtual venue. Similarly to the user management data table 1200, there is also a data table for managing parameters of non-human agents (NHAs) corresponding to 1203 to 1206 in the user management data table 1200. The table for managing NHAs is used to manage the participation form, display image, coordinates, orientation, etc. of NHAs in an event.

[0242] Figure 12 is a diagram showing examples of events for attracting and interacting with customers, combining examples with a large number of participants and examples with a small number of participants.

[0243] As shown in Figure 12(a), if the number of participants in an event is too small, participants will feel left out and will be less satisfied. On the other hand, as shown in Figure 12(c), if the number of participants in an event is too high, other participants will tend to invade your personal space, making you feel more pressured.

[0244] On the other hand, the present invention can eliminate the above-mentioned situation and realize an appropriate number (density) of participants in an event. In other words, by using the present invention, as shown in the example of Fig. 12(b), by increasing or decreasing the number of human agents and non-human agents according to the situation and controlling the number of people and agents in the event venue to fall within a certain range, it is possible to realize an appropriate number (density) of participants in an event, which is not too few so that participants do not feel alienated, nor too many so that participants feel oppressed.

[0245] It goes without saying that the object of the present invention can also be achieved by supplying a recording medium on which a program that realizes the functions of the above-mentioned embodiments is recorded to a system or device, and having the computer (or CPU or MPU) of that system or device read and execute the program stored on the recording medium.

[0246] In this case, the program itself read out from the recording medium will realize the novel functions of the present invention, and the recording medium storing the program will constitute the present invention.

[0247] Examples of recording media for supplying the program include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EPROMs, silicon disks, solid state drives, etc.

[0248] Furthermore, it goes without saying that the functions of the above-mentioned embodiments are not only realized by the computer executing a program it has read, but also includes cases where an OS (operating system) running on the computer performs some or all of the actual processing based on the instructions of the program, and the functions of the above-mentioned embodiments are realized through that processing.

[0249] Furthermore, it goes without saying that this also includes cases where a program read from a recording medium is written into a memory provided on a function expansion board inserted into a computer or a function expansion unit connected to a computer, and then a CPU or the like provided on the function expansion board or function expansion unit performs some or all of the actual processing based on the instructions of the program code, thereby realizing the functions of the above-mentioned embodiments.

[0250] Furthermore, by downloading and reading a program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or device can enjoy the effects of the present invention. Note that the present invention also includes configurations that combine the above-mentioned embodiments and their modified examples. [Explanation of symbols]

[0251] 10. Information processing terminal 20 Information processing equipment (PCs, servers, etc.) 30 Sensing Device 40 Physical Agent 41 Physical HA (HA: Human Agent) 42 Physical NHA (NHA: Non-Human Agent) 50 Communication networks (WAN (Wide Area Network), LAN (Local Area Network), etc.) 60 Physical Venues 61 Human 80 Virtual Venues 81 Virtual HA 82 Virtual NHA 90 Appropriate number of people

Claims

1. Control means for changing an object that operates by user operation to an object that operates independently of user operation, or changing an object that operates independently of user operation to an object that operates by user operation, based on the user's login status. An information processing device characterized by comprising:

2. Further comprising a means for receiving user login requests, The control means, upon receiving the login request, controls the object that operates independently of user actions to be changed to an object that operates based on actions by the user who made the login request. The information processing apparatus according to claim 1, characterized by the following:

3. Further comprising a means for receiving user logout requests, When the receiving means receives the logout request, the control means controls the object that operates based on the user who made the logout request to be changed to an object that operates independently of the user's operation. The information processing apparatus according to claim 1, characterized by the following:

4. The control means controls the object so as not to change information relating to at least one of the position, orientation, and design of the object when the object is modified. The information processing apparatus according to claim 1, characterized by the following:

5. The control means controls the object so as not to change its position and orientation, and to change the design of the object when the object is modified. The information processing apparatus according to claim 4, characterized by the above.

6. The control means controls the participation status of objects that operate independently of the user's operation based on total information of participants, including objects that operate by the user's operation and objects that operate independently of the user's operation, and target information which is a target value of the total information. The information processing apparatus according to claim 1, characterized by the following:

7. The control means is If the total information is less than the target information, control is made to increase the number of objects that operate independently of the user's actions. If the total information is greater than the target information, control the system to reduce the number of objects that operate independently of the user's actions. The information processing apparatus according to claim 6, characterized by the following:

8. The participant includes a user and an object that operates through user operations, The system further includes a means for calculating total information on users, objects operated by users, and objects that operate independently of user operations. The information processing apparatus according to claim 6, characterized by the following:

9. The total information is the density of participants and objects that operate independently of user actions within a predetermined area. The information processing apparatus according to claim 6, characterized by the following:

10. The total information is the number of participants using seats in a predetermined area and the number of objects that operate independently of user operations or the seat utilization rate in a predetermined area. The information processing apparatus according to claim 6, characterized by the following:

11. The event is an event that takes place in a virtual space and is at least one of the following: a festival, a music concert, or an academic presentation. An information processing apparatus according to any one of claims 6 to 10, characterized by the above.

12. The control means controls the object to be modified based on the user's login status to a predetermined mixed reality space. An information processing apparatus according to any one of claims 1 to 10, characterized by the following:

13. A method for controlling an information processing device, Control step by the control means of the information processing device, based on the user's login status, to change an object that operates by user operation to an object that operates independently of user operation, or to change an object that operates independently of user operation to an object that operates by user operation. A control method for an information processing device, characterized by comprising the following:

14. A program for causing at least one computer to function as each of the means of the information processing apparatus described in any one of claims 1 to 10.

Images