Virtual space monitoring apparatus, program, and virtual space monitoring system

Abstract

To provide a virtual space monitoring apparatus, a program, and a virtual space monitoring system for avoiding problems in a virtual space where avatars respectively corresponding to a plurality of users act.SOLUTION: A metaverse server 1 has a speech and behavior data acquiring unit 12 for acquiring speech and behavior data of avatars in a metaverse in which avatars respectively corresponding to a plurality of users act, a state estimation unit 13 for estimating states of the users corresponding to the avatars based on the speech and behavior data acquired by the speech and behavior data acquiring unit 12, and a performance change processing unit 14 for changing performance of the metaverse presented to the users in response to the states estimated by the state estimation unit 13.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a virtual space monitoring device, a program, and a virtual space monitoring system. [Background technology]

[0002] Currently, the metaverse, a virtual space on the Internet, is being utilized in a variety of fields. For example, there are cases where local governments are using the metaverse for educational purposes. In addition, metaverses that have functions such as those of stores and government offices are also emerging. In such a situation, problems such as violence and harassment that occur in the real world can occur in the Metaverse as well. In the educational field, for example, cases that develop into bullying apply, and in customer service applications, for example, cases of harassment by complainers apply.

[0003] In the field of games, playing in a virtual space with multiple players has been taking place before the use of the Metaverse. In games, there are systems that monitor the activities of users in the virtual space and penalize users who cause problems by forcibly logging them out or by restricting their functions (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2023-26335 Summary of the Invention [Problem to be solved by the invention]

[0005] The mechanism described in Patent Document 1 is implemented when a problem actually occurs. Furthermore, it is not appropriate to apply a mechanism that imposes penalties to a metaverse in the education field or for customer service purposes, and there is a need for a mechanism that can detect signs of problematic incidents before they occur and prevent them from occurring.

[0006] Therefore, an object of the present invention is to provide a virtual space monitoring device, program, and virtual space monitoring system for avoiding problems in a virtual space where avatars corresponding to multiple users are active. [Means for solving the problem]

[0007] The present invention solves the above problems by the following means. A first invention is a virtual space monitoring device comprising: a speech and behavior data acquisition means for acquiring speech and behavior data of avatars corresponding to each of a plurality of users in a virtual space in which the avatars are active; a state estimation means for estimating the state of the user corresponding to the avatar based on the speech and behavior data acquired by the speech and behavior data acquisition means; and a presentation change means for changing the presentation of the virtual space presented to the user in accordance with the state estimated by the state estimation means. A second invention is a virtual space monitoring device according to the first invention, wherein the state includes a normal state indicating a stable emotional state and an abnormal state different from the normal state, and the presentation change means, when the state estimated by the state estimation means is the abnormal state, changes the presentation of the virtual space presented to the user to one that changes the state of the user toward the normal state. A third invention is a virtual space monitoring device according to the second invention, wherein the abnormal state includes an excited state that is more excited than the normal state and a collapsed state that is more collapsed than the normal state, and wherein the presentation change means, when the state estimated by the state estimation means is the excited state, changes the presentation of the virtual space presented to the user to one that changes the state of the user in a direction that calms the state of the user, and when the state estimated by the state estimation means is the collapsed state, changes the presentation of the virtual space presented to the user to one that changes the state of the user in a direction that inspires the user. A fourth invention is a virtual space monitoring device according to the second invention, further comprising a state memory unit that stores the state estimated by the state estimation means in chronological order for each user, and the presentation change means compares the state estimated by the state estimation means with the state estimated in the past and stored in the state memory unit, and if the state is a change from the normal state to the abnormal state, changes the presentation of the virtual space to be presented to the user in accordance with the state, and if the state is a change from the abnormal state to the normal state, changes the presentation of the virtual space to be presented to the user in accordance with the state only if the normal state has continued for a predetermined period of time. A fifth invention is a virtual space monitoring device according to any one of the first to fourth inventions, wherein the speech and behavior data includes behavioral data of the avatar, and the state estimation means confirms the positional relationship of each avatar using the behavioral data of the avatar and the behavioral data of other avatars corresponding to other users, and when problematic behavior is confirmed from the change in the confirmed positional relationship, estimates the state of the user corresponding to the avatar. A sixth invention is a virtual space monitoring device according to the fifth invention, further comprising a history memory unit that stores the behavioral data of the avatar in chronological order for each user corresponding to the avatar, and the state estimation means estimates the state of the user corresponding to the avatar based on a result of comparing the behavioral data of the avatar with past behavioral data of the avatar stored in the history memory unit. A seventh invention is a virtual space monitoring device according to any one of the first to sixth inventions, wherein the speech and behavior data includes conversation data of the avatar, the conversation data includes speech data, and the state estimation means estimates the state of the user corresponding to the avatar based on the results of analyzing the speech data included in the conversation data of the avatar. An eighth invention is a virtual space monitoring device according to any one of the first to sixth inventions, wherein the speech and behavior data includes conversation data of the avatar, and the conversation data includes text data of the speech content, and the state estimation means estimates the state of the user corresponding to the avatar from the results of analyzing the text data included in the conversation data of the avatar. A ninth invention is a virtual space monitoring device in any one of the first to eighth inventions, wherein the performance change means changes at least one of the sound, lighting, texture, font, and additional characters of the virtual space depending on the state. A tenth invention is a virtual space monitoring device according to any one of the first to ninth inventions, further comprising an external environment control means for controlling the external environment of the user corresponding to the avatar in accordance with the state estimated by the state estimation means. An eleventh aspect of the present invention is a program for causing a computer to function as any one of the virtual space monitoring devices according to the first to tenth aspects of the present invention. A twelfth invention is a virtual space monitoring system comprising a terminal used by a user and a server communicatively connected to the terminal, wherein the server comprises: a speech and behavior data acquisition means for acquiring speech and behavior data of avatars corresponding to each of a plurality of users in a virtual space in which the avatars are active from the user's terminal; a state estimation means for estimating the state of the user corresponding to the avatar based on the speech and behavior data acquired by the speech and behavior data acquisition means; and a presentation change means for changing the presentation of the virtual space presented on the user's terminal in accordance with the state estimated by the state estimation means. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a virtual space monitoring device, program, and virtual space monitoring system for avoiding problems in a virtual space in which avatars corresponding to multiple users are active. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is an overall configuration diagram of a metaverse system according to an embodiment of the present invention and a functional block diagram of a metaverse server. [Figure 2] 10 is a diagram illustrating an example of an environment instruction table stored in a storage unit of the metaverse server according to the present embodiment. FIG. [Figure 3] 10 is a flowchart illustrating a metaverse monitoring process of the metaverse server according to the present embodiment. [Figure 4] FIG. 10 is a diagram showing an example of a screen displayed on the terminal according to the embodiment. [Figure 5] 10 is a flowchart illustrating a state estimation process of a metaverse server according to the present embodiment. [Figure 6] 10 is a flowchart showing the performance control processing of the metaverse server according to the present embodiment. [Figure 7] 10 is a flowchart illustrating a state update process of the metaverse server according to the present embodiment. [Figure 8] FIG. 10 is a diagram showing an example of a screen displayed on the terminal according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, this is merely an example, and the technical scope of the present invention is not limited to this example. (Embodiment) <Overall configuration of Metaverse System 100> FIG. 1 is a diagram showing the overall configuration of a metaverse system 100 according to this embodiment and a functional block diagram of a metaverse server 1. As shown in FIG. FIG. 2 is a diagram showing an example of the environment instruction table 22 stored in the storage unit 20 of the metaverse server 1 according to this embodiment.

[0011] 1, a metaverse system 100 (virtual space monitoring system) includes a metaverse server 1 (virtual space monitoring device, server) and a terminal 5. The metaverse server 1 and each user's terminal 5 are connected via a communication network N. In the metaverse system 100, a metaverse server 1 provides a metaverse, which is a virtual space in which users participate. In the metaverse system 100, the metaverse server 1 acquires the behavior of an avatar active in the metaverse corresponding to the user, estimates the user's state from the behavior of the avatar, and changes the presentation of the metaverse according to the estimated state. In this way, problems between users in the metaverse can be avoided before they arise.

[0012] <Metaverse Server 1> The metaverse server 1 is a device that provides a metaverse to users and monitors the activities of avatars corresponding to the users. The metaverse server 1 is, for example, a server, and performs processing via communication with the terminal 5. The metaverse server 1 may also be, for example, a general personal computer (PC) or the like. The metaverse server 1 includes a control unit 10, a storage unit 20, and a communication interface unit 29.

[0013] The control unit 10 is a CPU (Central Processing Unit) that controls the entire metaverse server 1. The control unit 10 appropriately reads and executes an OS (Operating System) and various application programs stored in the storage unit 20, thereby cooperating with the above-mentioned hardware and executing various functions. Before describing the control unit 10, the storage unit 20 will be described.

[0014] The storage unit 20 is a storage device such as a hard disk or semiconductor memory element for storing programs, data, etc. required for the operation of the metaverse server 1. The storage unit 20 includes a program storage unit 21, an environment instruction table 22, a state storage unit 23, and a history storage unit 24. The program storage unit 21 is a storage area for storing programs, and stores a metaverse program 21a, a user monitoring program 21b, and an environment control program 21c.

[0015] The metaverse program 21a is, for example, a program for executing the functions of the metaverse processing unit 11 of the control unit 10, which will be described next. The user monitoring program 21b is a program for executing the functions of, for example, the speech and behavior data acquisition unit 12 and the state estimation unit 13 of the control unit 10, which will be described next. The environmental control program 21c is a program for executing the functions of the performance change processing unit 14 and the external environment processing unit 15 of the control unit 10, which will be described next, for example. The configuration of each program is not limited to this. For example, the various functions of the control unit 10 may be executed by a single program, or the user monitoring program 21b and the environment control program 21c may be executed as a single program, or other configurations may be used.

[0016] The environment instruction table 22 is a table that associates the user's state with the user's environment, including the metaverse corresponding to the state. FIG. 2 shows an example of the environment instruction table 22. The state is a state related to the user's emotions, and in this example, is shown as one of three states: "normal," "excited," and "depressed." "Normal" indicates a state in which the user's emotions are stable. "Excitement" indicates an abnormal state (excited state) in which the user's emotions are higher than the "normal" state. "Depression" indicates an abnormal state (depressed state) in which the user's emotions are more depressed than in a "normal" state.

[0017] The environment is broadly divided into the metaverse environment and the external environment, which is the environment outside the metaverse. In this example, the Metaverse environment includes sound, lighting, texture, font (character style), and presentation (additional characters). Sound refers to the sounds of the Metaverse, including background music and ambient sounds. The aim of stereophonic sound is to enhance the sense of realism when playing background music and the like, and to actively draw the attention of users in a negative emotional state to the sound, thereby encouraging them to change their mood. Techniques for creating stereophonic sound can be realized using known techniques, such as those disclosed in Japanese Patent Application Laid-Open No. 2007-158985. Lighting is about the brightness of the Metaverse. Texture refers to the texture of furnishings and other items in the Metaverse.

[0018] Fonts refer to the typeface, size, and color of text messages exchanged in the metaverse, such as through chat. For example, for "excitement," the font may be rounded, smaller in size, and lighter in color. For "depression," the font may be angular, with decorations such as outlines and shadows, larger in size, and brighter in color. In this way, by changing the font depending on the state, the user's mood can be guided to a neutral state. The effects are things that appear in the metaverse. For example, a scene that shows a smiling self reflected in a mirror is intended to stimulate the user's mind.

[0019] The external environment refers to the user's actual environment, and includes, for example, scent, vibration, temperature, and the like. The scent is emitted from a device such as an aroma diffuser linked to the terminal 5. The vibration is generated from a device linked to the terminal 5, such as a smart watch. The temperature is generated by a device such as a fan or heater connected to the terminal 5. The above-mentioned examples of environments according to the states are merely examples. Moreover, what has been described here as the environment instruction table 22 may be, for example, an internal table of the environment control program 21c.

[0020] The state storage unit 23 shown in FIG. 1 is a storage area that stores the state of each user estimated by the control unit 10. The state storage unit 23 stores, for example, user states including the current state in chronological order from the past state, for each user ID (ID) that identifies each user. The history storage unit 24 is a storage area for storing speech and behavior data of avatars. The history storage unit 24 stores, for example, in chronological order, speech and behavior data of an avatar corresponding to the user, associated with the user ID. The speech and behavior data includes avatar behavior data and conversation data. Here, the avatar behavior data can be represented, for example, by the time the behavior occurred, the avatar's position (XY coordinates), and the avatar's orientation (directional vector). Furthermore, the avatar conversation data can be represented, for example, by the time the conversation occurred, and text data or spoken voice data.

[0021] It should be noted that the items stored in the storage unit 20 are not limited to these. For example, the storage unit 20 may have a storage area for storing user information. The storage area for storing user information may store information related to the user, such as a user ID and a login password, as well as device IDs for identifying various external devices (not shown) used as the user's external environment.

[0022] Next, the control unit 10 will be described. The control unit 10 includes a metaverse processing unit 11, a speech and behavior data acquisition unit 12, a state estimation unit 13, a performance change processing unit 14, and an external environment processing unit 15. When a connection is made from terminal 5 to metaverse server 1, metaverse processing unit 11 outputs an initial metaverse screen (not shown) to terminal 5. Furthermore, metaverse processing unit 11 performs various processes for realizing the metaverse. The various processes for realizing the metaverse will be described later.

[0023] The speech and behavior data acquisition unit 12 functions as a speech and behavior data acquisition means. The speech and behavior data acquisition unit 12 acquires speech and behavior data of an avatar from the terminal 5. The avatar's speech and behavior data can also be said to be speech and behavior data of a user who operates the avatar using the terminal 5. The speech and behavior data acquisition unit 12 then stores the acquired speech and behavior data in the history storage unit 24. The state estimation unit 13 functions as a state estimation means. The state estimation unit 13 estimates the state of the user corresponding to the avatar based on the speech and behavior data acquired by the speech and behavior data acquisition unit 12. The user state estimated by the state estimation unit 13 is a state related to the user's emotions based on the speech and behavior of the avatar, and is indicated by three, for example, "normal," "excited," and "depressed."

[0024] The state estimation unit 13 checks the positional relationship between avatars using the behavioral data of the avatar and other avatars, and when problematic behavior is confirmed from the transition of the confirmed positional relationship, estimates the state of the user corresponding to the avatar. Problematic behavior here includes, for example, behaviors such as "stalking" and "overhanging."

[0025] For example, the state estimation unit 13 determines that "stalking" is occurring when the following behaviors (1) to (3) continue for a certain period of time. (1) Another user's avatar is within a certain distance of the user's avatar. (2) The body of the other user's avatar is facing the direction of the user's avatar, and the user's avatar is not facing the direction of the other user's avatar. (3) The user's avatar moves away from the other users' avatars, and the other users' avatars move toward the user's avatar.

[0026] If the state estimation unit 13 determines that the other user is "stalking," it determines that there is a high possibility that the other user will behave aggressively, and estimates the state of the other user as "excited." In addition, "overhanging" refers to an action in which another user's avatar hangs over the user's avatar, blocking the view of the user's avatar. In this case, the state estimation unit 13 also determines that the other user is likely to behave aggressively, and estimates the state of the other user as "excited."

[0027] Furthermore, the state estimation unit 13 estimates the state of the user corresponding to the avatar from the result of comparing the behavior data of the avatar with past behavior data of the avatar stored in the history storage unit 24. The state estimation unit 13 aggregates the past behavioral data of the avatar stored in the history storage unit 24, and calculates three indices, for example, the ratio of the frequency of stationary and moving periods (number of times moving / number of times stationary, etc.), the average speed of movement, and the change in orientation (body rotation), and regards the long-term average of these as the standard behavioral pattern of the avatar. Next, the state estimation unit 13 similarly calculates the three indices from the behavioral data of the avatar acquired by the speech and behavior data acquisition unit 12. Then, if the indices for the immediately preceding fixed period differ from the standard behavioral pattern of the avatar, the state estimation unit 13 estimates that the state of the user corresponding to the avatar is abnormal. For example, if the movement in the immediately preceding fixed period was too vigorous, the state estimation unit 13 estimates that the user's state is "excited," and if the movement in the immediately preceding fixed period was too little, the state estimation unit 13 estimates that the user's state is "depressed."

[0028] Furthermore, when the conversation data of the avatar is speech voice data, the state estimation unit 13 estimates the state of the user corresponding to the avatar based on the result of analyzing the speech voice data included in the conversation data of the avatar. The state estimation unit 13 estimates the user's state using, for example, a technique for estimating emotion from intonation or voice of uttered voice data. As a technique for estimating emotion from voice, for example, the techniques described in U.S. Patent No. 6,638,217 and U.S. Patent No. 7,165,033 can be used. When this technique is used, the state estimation unit 13 estimates the user's state as "excited" when the index "emotional" is equal to or greater than a specified value, estimates the user's state as "depressed" when the index "anxiety" is equal to or greater than a specified value, and estimates the user's state as "normal" in all other cases.

[0029] Furthermore, when the conversation data of the avatar is text data of speech content, the state estimation unit 13 estimates the state of the user corresponding to the avatar from the results obtained by analyzing the text data included in the conversation data of the avatar. The state estimation unit 13 can estimate the state of the user corresponding to the avatar from specific words and the usage of the specific words, for example, as a result obtained by analyzing text data. Here, the specific words can be realized by storing, for example, keywords corresponding to "excitement" or "depression" in the storage unit 20 in advance. Here, keywords corresponding to "excitement" include aggressive words, words related to anger, negative words, etc., and keywords corresponding to "depression" include self-deprecating words, words related to sadness, etc. Then, the state estimation unit 13 estimates that the user is "excited" or "depressed" when a corresponding keyword is detected a predetermined number of times or more within a predetermined time. The state estimation unit 13 may estimate the state not only based on the word but also by checking the usage of the word sequence, etc. For example, when the word "kill" is registered, it will not be estimated as "excitement" if it is used in a restrained manner, such as "holding one's breath."

[0030] Furthermore, the state estimation unit 13 can extract words expressing the user's emotions by, for example, performing natural language processing on the text data from the context of the text as semantic analysis, and can estimate the user's state from the extracted words. For example, the technology described in JP 2019-53460 A can be used as a technology for estimating emotions from text data. When using this technology, the user's emotions estimated from the extracted words can be obtained as, for example, the degree of joy, anger, sadness, or happiness. Therefore, the state estimation unit 13 estimates the user's emotions as "excited" when the index "anger" is equal to or greater than a specified value, and estimates the user's state as "depressed" when the index "sadness" is equal to or greater than a specified value.

[0031] The presentation change processing unit 14 functions as a presentation change unit. The presentation change processing unit 14 changes the presentation of the metaverse presented to the user in accordance with the state estimated by the state estimation unit 13. When the state estimated by the state estimation unit 13 is an abnormal state, the presentation change processing unit 14 changes the metaverse presented to the user to a presentation for changing the user's state toward a normal state. More specifically, when the state estimated by the state estimation unit 13 is "excited," the presentation change processing unit 14 changes the metaverse presented to the user to a presentation for changing the user's state toward a calmer state, and when the state estimated by the state estimation unit 13 is "depressed," the presentation change processing unit 14 changes the metaverse presented to the user to a presentation for changing the user's state toward an excited state.

[0032] Furthermore, the state estimated by state estimation unit 13 is compared with the state of the same user estimated in the past and stored in state storage unit 23, and if the state changes from "normal" to "excited" or "depressed", presentation change processing unit 14 changes the presentation of the metaverse presented to the user in accordance with the state. On the other hand, if the state changes from "excited" or "depressed" to "normal", presentation change processing unit 14 changes the presentation of the metaverse presented to the user to "normal" only if the state estimated by state estimation unit 13 has been "normal" for a predetermined consecutive period of time. Here, the presentation change processing unit 14 changes at least one of the sound, lighting, texture, font, and presentation of the metaverse to correspond to the changed state of the user.

[0033] The effect change processing unit 14 refers to the environment instruction table 22 and transmits condition change data for changing various environmental control conditions corresponding to the state to the terminal 5. The condition change data includes at least one of data related to sound control, changes in lighting and texture of CG (Computer Graphics) for screen rendering, fonts (typeface, size, color, etc.), addition of effects, etc. The presentation change processing unit 14 may send, as condition change data, for example, a set of content with the metaverse presentation changed to the terminal 5, or may send parameters for changing the metaverse presentation to the terminal 5, and the metaverse environment may be changed on the terminal 5 using the received parameters.

[0034] The external environment processing unit 15 functions as an external environment control means. The external environment processing unit 15 controls the external environment of the user corresponding to the avatar according to the state estimated by the state estimation unit 13. The external environment processing unit 15 refers to the environment instruction table 22 and transmits control data for changing various environmental control conditions corresponding to the state to, for example, an external device corresponding to the user. Examples of control data for changing various environmental control conditions include an instruction signal for generating or stopping a fragrance, an instruction signal for generating vibrations, changing or stopping the vibration frequency, and data for raising or lowering the temperature. The details of each of these functions will be described later. The communication interface unit 29 is an interface for communicating with each user's terminal 5 and the like via the communication network N.

[0035] It should be noted that a computer refers to an information processing device equipped with a control unit, a storage device, etc., and the metaverse server 1 is an information processing device equipped with a control unit 10, a storage unit 20, etc., and is included in the concept of a computer. Furthermore, there is no limit to the number of pieces of hardware that make up the metaverse server 1. It may be configured with one or more pieces as needed. Furthermore, the hardware of the metaverse server 1 may be configured to include various servers such as a web server, a database server, and an application server as needed, and may be configured with one server or with separate servers. Furthermore, the metaverse server 1 may be, for example, a cloud.

[0036] <Terminal 5> Terminal 5 is a computer used by a user participating in the metaverse. Terminal 5 can be configured, for example, as a PC, tablet terminal, smartphone, etc. Terminal 5 may also be an HMD (head mounted display), etc. Although not shown, terminal 5 includes a control unit, a memory unit, a display unit, an input unit, a communication interface unit, etc. The display unit and input unit may be integrated into one unit, such as a touch panel display. For example, when terminal 5 receives a set of content with a modified metaverse presentation, it uses the received set of content to perform processing such as playback. Also, when terminal 5 receives parameters for modifying the metaverse presentation, it generates content using the parameters and performs processing such as playback.

[0037] <Communication Network N> The communication network N is a network for communication between the metaverse server 1, the terminal 5, and an external device (not shown). The communication network N is, for example, a communication network such as the Internet, and may be wired or wireless.

[0038] <Processing Description> Next, the processing performed by the metaverse server 1 will be described. FIG. 3 is a flowchart showing the metaverse monitoring process of the metaverse server 1 according to this embodiment. FIG. 4 is a diagram showing an example of a screen displayed on the terminal 5 according to this embodiment.

[0039] When terminal 5 accesses metaverse server 1, in step S (hereinafter, "step S" will be simply referred to as "S") 11 of Fig. 3, control unit 10 of metaverse server 1 detects the access from terminal 5. Thereafter, control unit 10 of metaverse server 1 performs, for example, authentication of the user of terminal 5. If authentication is successful, in S12, the metaverse processing unit 11 transmits an avatar corresponding to the user of terminal 5 and metaverse data to terminal 5. At that time, the metaverse processing unit 11 refers to the environment instruction table 22 and transmits metaverse data of the environment when the state is "normal" to terminal 5. The metaverse processing unit 11 may transmit a set of metaverse content to terminal 5, or may transmit parameters for generating the metaverse to terminal 5.

[0040] In the terminal 5, the control unit displays the metaverse using the received data. FIG. 4 shows an example of a metaverse screen 60 displayed on the display unit of the terminal 5. The metaverse screen 60 is in a "normal" state, for example, with standard lighting. Then, the metaverse processing unit 11 of the metaverse server 1 and the control unit of the terminal 5 perform processing to realize the metaverse. More specifically, the control unit of terminal 5 displays the metaverse and plays audio data using data acquired from metaverse server 1. Users view audiovisual information presented on terminal 5, and at the same time, they are active in the metaverse by moving avatars and chatting (by voice and / or text) via terminal 5. Furthermore, by multiple users simultaneously connecting to metaverse server 1 from their respective terminals 5, avatars corresponding to the multiple users can interact in the metaverse.

[0041] The metaverse processing unit 11 of the metaverse server 1 receives behavioral data related to the movement of each user from the number of connected terminals 5, and moves the position of the avatar corresponding to each user within the metaverse. At the same time, the metaverse processing unit 11 receives chat conversation data from each terminal 5. The metaverse processing unit 11 then distributes the behavioral data and conversation data of the avatar corresponding to each user to each user's terminal 5. The metaverse processing unit 11 performs processing to transmit a set of metaverse content and parameters for updating the metaverse as necessary.

[0042] In the following, the processing relating to user monitoring will be mainly explained, excluding the above-mentioned processing for reproducing the metaverse, etc. In S13 of FIG. 3, the speech and behavior data acquisition unit 12 acquires speech and behavior data from the terminal 5. In S14, the speech and behavior data acquisition unit 12 stores the acquired speech and behavior data in the history storage unit 24. In S15, the control unit 10 performs a state estimation process, which will be described later. In S16, the control unit 10 performs a performance control process, which will be described later. In S17, the control unit 10 determines whether or not to terminate the processing for the terminal 5. For example, when the user of the terminal 5 logs out from the metaverse server 1 and all users participating in the metaverse have left due to logging out or the like, the control unit 10 determines to terminate the processing for the terminal 5. If the processing for the terminal 5 is to be terminated (S17: YES), the control unit 10 terminates this processing. On the other hand, if the processing for the terminal 5 is not to be terminated (S17: NO), the control unit 10 moves the processing to S13.

[0043] Next, the state estimation process will be described. FIG. 5 is a flowchart showing the state estimation process of the metaverse server 1 according to this embodiment. In S31 of FIG. 5, the state estimation unit 13 compares the behavior data of the avatar corresponding to the user with the past behavior data of the avatar, and estimates the state of the user. In S32, the state estimation unit 13 checks whether or not problematic behavior is occurring based on the positional relationship between the avatar corresponding to the user in the metaverse and the avatars of other users, and estimates the state of the user. In S33, the state estimation unit 13 determines whether the conversation of the avatar corresponding to the user is voice. If it is voice (S33: YES), the state estimation unit 13 proceeds to S34. On the other hand, if it is not voice, that is, if it is text (S33: NO), the state estimation unit 13 proceeds to S35.

[0044] In S34, the state estimation unit 13 estimates the state of the user based on the result of analyzing the voice data, and then the state estimation unit 13 moves the process to S36. In S35, the state estimation unit 13 estimates the state of the user from the results obtained by analyzing the text data. If the conversation of the avatar corresponding to the user is voice, the state estimation unit 13 may convert the voice data into text and perform the process of S35 after the process of S34.

[0045] In S36, the state estimation unit 13 performs a determination process related to the estimated state. In the above, the user's state is estimated using behavioral data and conversational data as language data, but it is assumed that the estimation results may differ. Therefore, when the user's states differ, the state estimation unit 13 determines a state other than "normal" with priority, for example. This is to lower the priority of the "normal" state in order to prevent problems from occurring. Furthermore, the state estimation unit 13 may, for example, prioritize the user's state based on conversational data over the user's state based on behavioral data. Thereafter, the state estimation unit 13 proceeds to S16 in FIG. 3.

[0046] Next, the performance control process will be described. FIG. 6 is a flowchart showing the performance control process of the metaverse server 1 according to this embodiment. FIG. 7 is a flowchart showing the state update process of the metaverse server 1 according to this embodiment. FIG. 8 is a diagram showing an example of a screen displayed on the terminal 5 according to this embodiment. In S51 of FIG. 6, the control unit 10 performs a state update process.

[0047] Here, the state update process will be described with reference to FIG. 7, the state estimation unit 13 determines whether the estimated state is "excitement." If the estimated state is "excitement" (S61: YES), the state estimation unit 13 proceeds to S62. On the other hand, if the estimated state is not "excitement" (S61: NO), the state estimation unit 13 proceeds to S63. In S62, the state estimation unit 13 sets the current state to "excited" and registers this in the state storage unit 23. After that, the state estimation unit 13 moves the process to S52 in FIG.

[0048] In S63, the state estimation unit 13 determines whether the estimated state is a "drop." If the estimated state is a "drop" (S63: YES), the state estimation unit 13 proceeds to S64. On the other hand, if the estimated state is not a "drop" (S63: NO), the state estimation unit 13 proceeds to S65. In S64, the state estimation unit 13 sets the current state to "drop" and registers this in the state storage unit 23. Thereafter, the state estimation unit 13 shifts the process to S52 in FIG.

[0049] In S65, the state estimation unit 13 determines whether the estimated state is "normal" and has continued for a predetermined time. The predetermined time may be, for example, a few minutes. If the estimated state is "normal" and has continued for the predetermined time (S65: YES), the state estimation unit 13 proceeds to S66. On the other hand, if the estimated state is "normal" but has not continued for the predetermined time (S65: NO), the state estimation unit 13 proceeds to S52 in FIG. 6. Note that, in order for the state estimation unit 13 to determine whether the estimated state is "normal" and has continued for the predetermined time, it is necessary to record that the current estimated state is "normal." Therefore, the state estimation unit 13 may store the estimated state and its time together with the user ID in a temporary storage area of ​​the storage unit 20, for example. In S66, the state estimation unit 13 sets the current state to "normal" and registers it in the state storage unit 23. Thereafter, the state estimation unit 13 moves the process to S52 in FIG.

[0050] In S52 of Fig. 6, the control unit 10 determines whether or not there has been a change in the current state. If there has been a change in the current state (S52: YES), the control unit 10 proceeds to S53. On the other hand, if there has been no change in the current state (S52: NO), the control unit 10 proceeds to S17 of Fig. 3. In S53, the performance change processing unit 14 transmits condition change data for changing the environmental control conditions of the metaverse based on the current state to the terminal 5 of the user corresponding to the state.

[0051] Through this process, the control unit of the user terminal 5 reflects the change condition data received from the metaverse server 1 in the metaverse. FIG. 8 shows examples of metaverse screens 61 to 63 in which the changed condition data is reflected in the metaverse. The metaverse screen 61 shown in Figure 8(A) is an example when the user's state is "normal." The metaverse screen 62 shown in Figure 8(B) is an example when the user's state is "excited." The metaverse screen 63 shown in Figure 8(C) is an example when the user's state is "depressed." The metaverse screen 62 shows darker lighting than the metaverse screen 61. By changing the metaverse in this way, an excited user can be encouraged to regain their composure. Furthermore, the metaverse screen 63 shows brighter lighting than the metaverse screen 61. By changing the metaverse in this way, it is possible to encourage a depressed or distracted user to pull themselves together.

[0052] 6, the external environment processing unit 15 transmits control data for the external device based on the current state to the external device used by the user. This allows changes to be made to the user's real environment, such as scent, vibration, temperature, etc. After that, the control unit 10 proceeds to S17 in FIG.

[0053] As described above, according to this embodiment, the metaverse server 1 has the following effects. (1) We acquire speech and behavior data of avatars in the metaverse where avatars corresponding to multiple users are active, estimate the state of the user corresponding to the avatar based on the acquired speech and behavior data, and change the metaverse presentation presented to the user according to the estimated state. Therefore, by inferring the user's state from the avatar's behavior, it is possible to detect signs of a problematic incident by the user, and by changing the presentation of the metaverse according to the estimated user's state, it is possible to provide a mechanism for guiding the user's emotions in a neutral direction. As a result, it is possible to maintain a peaceful state in the metaverse, and it is possible to effectively achieve the goals of the metaverse, such as obtaining good educational effects and promoting smooth customer service and interaction.

[0054] (2) The state includes a normal state, which indicates a stable emotional state, and an abnormal state that is different from the normal state. When the estimated state is an abnormal state, the metaverse presented to the user is changed to one that changes the user's state toward the normal state. Therefore, by changing the metaverse to produce a performance that changes the estimated user's state toward a normal state, the user's emotions can be guided toward maintaining calm.

[0055] (3) Abnormal states include "excitement," which is an excited state that is higher than the normal state, and "depression," which is a depressed state that is lower than the normal state. When the estimated state is "excitement," the metaverse presented to the user is changed to a presentation that will change the user's state in a calming direction. When the estimated state is "depression," the metaverse presented to the user is changed to a presentation that will change the user's state in a more inspiring direction. Therefore, in order to change the estimated user's state toward a normal state, if the user is "excited" it can be made to be calming, and if the user is "depressed" it can be made to be motivating.

[0056] (4) The estimated state is compared with previously estimated states stored in the state memory unit 23, which stores estimated states for each user in chronological order. If the state is a change from a normal state to an abnormal state, the metaverse presentation presented to the user is changed according to the state. If the state is a change from an abnormal state to a normal state, the metaverse presentation presented to the user is changed according to the state only if the state remains normal for a predetermined consecutive period of time. Therefore, if a sign of a problematic incident by a user is detected, the presentation of the metaverse can be changed immediately, and if the problematic incident no longer occurs, the presentation of the metaverse can be changed after checking the progress.

[0057] (5) The behavioral data includes behavioral data of the avatar, and the positional relationship of each avatar is confirmed using the behavioral data of the avatar and the behavioral data of other avatars corresponding to other users.If problematic behavior is confirmed from the changes in the confirmed positional relationship, the state of the user corresponding to the avatar is estimated. Therefore, problematic behavior can be detected from the behavior of each avatar. Furthermore, when problematic behavior is confirmed, the user's state can be estimated assuming that a problematic incident has been detected.

[0058] (6) The state of the user corresponding to the avatar is estimated from the results of comparing the avatar's behavioral data with past avatar behavioral data stored in the history memory unit 24, which stores avatar behavioral data for each user in chronological order. Therefore, it is possible to detect whether or not the user is behaving in a way that is different from usual, and the state of the user can be estimated based on the detection.

[0059] (7) The speech and behavior data includes avatar conversation data, and the conversation data includes speech data, and the state of the user corresponding to the avatar is estimated based on the results of analyzing the speech data included in the avatar conversation data. Therefore, the state of the user can be estimated from the voice uttered by the user.

[0060] (8) The speech and behavior data includes avatar conversation data, which includes text data of the speech content, and the state of the user corresponding to the avatar is estimated from the results of analyzing the text data included in the avatar conversation data. Therefore, the user's state can be estimated from the text of the user's utterance.

[0061] (9) At least one of the sound, lighting, texture, font, and presentation of the Metaverse is changed depending on the state. This allows for a performance that appeals to the user's visual and auditory senses, which can serve as an opportunity for the user to become aware of their own emotional state.

[0062] (10) The external environment of the user corresponding to the avatar is controlled according to the estimated state. Therefore, for example, it is possible to create an effect that appeals to the user's sense of smell or touch, which may serve as an opportunity for the user to become aware of their own emotional state.

[0063] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. Furthermore, the effects described in the embodiments are merely a list of the most preferable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments. Note that the above-described embodiments and the modified embodiments described below can be used in appropriate combinations, but detailed description thereof will be omitted.

[0064] (Variations) (1) In the present embodiment, the user's state is estimated as one of three states: "normal," "excited," and "depressed." However, the present invention is not limited to this. It is sufficient if a negative state can be estimated as a state other than "normal," and other states may also be used. Furthermore, the environment instruction table 22 is an example of a metaverse presentation that guides the user to a "normal" state when a negative state is estimated.

[0065] (2) In the present embodiment, data transmission to an external device is performed from a metaverse server, but this is not limiting. Data may be transmitted to an external device via a terminal used by a user.

[0066] (3) In the present embodiment, examples of analysis of voice data, analysis of text data, and analysis of behavioral history have been described, but the present invention is not limited to these methods. Analysis may be performed by other methods.

[0067] (4) In the present embodiment, the metaverse presentation change is applied to the device of the user whose state is estimated, but this is not limiting. For example, the metaverse presentation change may be applied to the devices of users corresponding to other avatars in the same metaverse, or to the devices of users corresponding to avatars in the same metaverse that are located a certain distance from the avatar corresponding to the user whose state is estimated.

[0068] (5) Although not specifically described in this embodiment, a mechanism may be added to the metaverse by having a counseling avatar appear near the target user's avatar and interviewing the target user using an AI dialogue function. For example, when a user's condition is abnormal, the counseling avatar approaches the avatar corresponding to the user. The counseling avatar then speaks to the avatar corresponding to the user, allowing the two avatars to engage in a conversation. The content of the interview is summarized, for example, using the summarization function of the generation AI, and the interview content is notified to an online support staff member operating the metaverse. For the target user, since the other party is an AI, it is easier for them to honestly communicate their situation and feelings without being wary. On the other hand, for the online support staff, the primary investigation function reduces their burden and organizes the situation, thereby benefiting from more accurate support.

[0069] (6) Although not specifically mentioned in this embodiment, for example, a function for adjusting language may be added. As an example, a mechanism may be added in which the user's hometown or other attributes are registered, and after converting utterances from other avatars into text, the attributes are referenced to convert the text into the user's native language or dialect, and the resulting text is played back using voice synthesis. Conversion into the user's native language or dialect can be performed, for example, using a generation AI. Furthermore, the tone of voice may be converted to, for example, a tone of voice that has a calming effect on the other person, such as that of an elderly woman, and then output. [Explanation of symbols]

[0070] 1 Metaverse Server 5. Terminal 10 Control Unit 11 Metaverse Processing Unit 12 Behavioral Data Acquisition Department 13 State estimation unit 14. Production change processing section 15 External Environment Processing Department 20 Memory section 21a Metaverse Program 21b User Monitoring Program 21c Environmental Control Program 22 Environmental Instruction Table 23 State memory unit 24 History memory section 60, 61, 62, 63 Metaverse screen 100 Metaverse System N Communication Network

Claims

1. a speech and behavior data acquisition means for acquiring speech and behavior data of avatars corresponding to each of a plurality of users in a virtual space in which the avatars are active; a state estimation means for estimating a state of the user corresponding to the avatar based on the speech and behavior data acquired by the speech and behavior data acquisition means; a presentation change means for changing a presentation of the virtual space presented to the user in accordance with the state estimated by the state estimation means; A virtual space monitoring device comprising:

2. 2. The virtual space monitoring device according to claim 1, The state includes a normal state indicating a stable emotional state and an abnormal state different from the normal state, The presentation change means, when the state estimated by the state estimation means is the abnormal state, changes the presentation of the virtual space presented to the user to one that changes the user's state toward the normal state, in a virtual space monitoring device.

3. 3. The virtual space monitoring device according to claim 2, The abnormal state includes a heightened state that is higher than the normal state and a collapsed state that is lower than the normal state, The presentation change means, when the state estimated by the state estimation means is the excited state, changes the presentation of the virtual space presented to the user to one that will change the user's state in a calming direction, and when the state estimated by the state estimation means is the collapsed state, changes the presentation of the virtual space presented to the user to one that will change the user's state in an inspiring direction, a virtual space monitoring device.

4. 3. The virtual space monitoring device according to claim 2, a state storage unit that stores the state estimated by the state estimation means in chronological order for each user; The presentation change means compares the state estimated by the state estimation means with the state estimated in the past and stored in the state memory unit, and if the state is a change from the normal state to the abnormal state, changes the presentation of the virtual space presented to the user in accordance with the state, and if the state is a change from the abnormal state to the normal state, changes the presentation of the virtual space presented to the user in accordance with the state only if the normal state remains for a predetermined consecutive period of time, in a virtual space monitoring device.

5. 2. The virtual space monitoring device according to claim 1, the speech and behavior data includes behavior data of the avatar; The state estimation means confirms the positional relationship of each avatar using the behavioral data of the avatar and the behavioral data of other avatars corresponding to other users, and when problematic behavior is confirmed from the changes in the confirmed positional relationship, estimates the state of the user corresponding to the avatar.

6. 6. The virtual space monitoring device according to claim 5, a history storage unit that stores the behavioral data of the avatar in chronological order for each user corresponding to the avatar; The state estimation means estimates the state of the user corresponding to the avatar from the results of comparing the behavioral data of the avatar with past behavioral data of the avatar stored in the history memory unit.

7. 2. The virtual space monitoring device according to claim 1, the speech and behavior data includes conversation data of the avatar; the conversation data includes speech data, The state estimation means estimates the state of the user corresponding to the avatar based on a result of analyzing the speech data included in the conversation data of the avatar.

8. 2. The virtual space monitoring device according to claim 1, the speech and behavior data includes conversation data of the avatar; the conversation data includes text data of the conversation content, The state estimation means estimates the state of the user corresponding to the avatar from a result of analyzing the text data included in the conversation data of the avatar.

9. 2. The virtual space monitoring device according to claim 1, The presentation change means changes at least one of the sound, lighting, texture, font, and additional characters of the virtual space in accordance with the state of the virtual space monitoring device.

10. 2. The virtual space monitoring device according to claim 1, The virtual space monitoring device includes an external environment control means for controlling an external environment of the user corresponding to the avatar in accordance with the state estimated by the state estimation means.

11. Computer, a speech and behavior data acquisition means for acquiring speech and behavior data of avatars corresponding to each of a plurality of users in a virtual space in which the avatars are active; a state estimation means for estimating a state of the user corresponding to the avatar based on the speech and behavior data acquired by the speech and behavior data acquisition means; a presentation change means for changing a presentation of the virtual space presented to the user in accordance with the state estimated by the state estimation means; A program to make it function as such.

12. a terminal used by a user; a server communicably connected to the terminal; A virtual space monitoring system comprising: The server a speech and behavior data acquisition means for acquiring speech and behavior data of avatars corresponding to a plurality of users in a virtual space in which the avatars are active from the terminals of the users; a state estimation means for estimating a state of the user corresponding to the avatar based on the speech and behavior data acquired by the speech and behavior data acquisition means; a presentation change means for changing a presentation of the virtual space presented on the terminal of the user in accordance with the state estimated by the state estimation means; A virtual space monitoring system comprising:

Images