Emotion analysis method in mr system, and mr system

Abstract

Provided is an emotion analysis method in a mixed reality (MR) system, said emotion analysis method comprising: a first step for using an external camera to capture images of a user of an MR headset and acquiring a video; a second step for extracting a person from the video; a third step for detecting data of the extracted person; a fourth step for performing emotion analysis of the user on the basis of the data of the detected person; and a fifth step for displaying, on the display of the MR headset, a representation based on the result of the emotion analysis.

Description

Emotion analysis method in MR system and MR system

[0001] The present invention relates to an emotion analysis method in an MR system and an MR system.

[0002] Mixed reality (MR) applications are becoming popular, and virtual idols, virtual tour guides, and the like are becoming available.

[0003] To provide a new MR system.

[0004] The following configuration is provided as an example.

[0005] [1] A method for emotion analysis in a mixed reality (MR) system, comprising: a first step of capturing an image of a user of an MR headset and a spatial environment with a camera to acquire the image; a second step of extracting a person from the image; a third step of detecting data of the extracted person; a fourth step of analyzing the emotion and body posture of the user based on the data of the detected person; and a fifth step of displaying the results of the analysis on a display of the MR headset.

[0006] [2] The method according to [1], wherein in the second step, people are extracted by detecting objects from the video, determining whether each of the detected objects is a dynamic object or a static object, and determining whether each of the dynamic objects is a person.

[0007] [3] The method according to [2], wherein the video includes a plurality of frames, and in the second step, it is determined whether each of the detected objects is a dynamic object or a static object by comparing the plurality of frames.

[0008] [4] The method according to any one of [1] to [3], wherein in the third step, a person's pattern is detected based on whether the person matches a predetermined pattern, and in the fourth step, an emotion analysis of the user is performed based on the person's pattern.

[0009] [5] The method according to any one of [1] to [4], wherein the fourth step includes: analyzing the data of the person from the MR headset; analyzing the emotion and the body posture by an analysis server; and transmitting the results of the analysis from the analysis server and displaying them on the MR headset.

[0010] [6] A mixed reality (MR) system comprising: an external camera that captures an image of a user of an MR headset and generates an image; an analysis server that extracts a person from the image, detects data of the extracted person, and analyzes the user's emotions and body posture based on the captured image; and an MR headset display that displays a representation based on the results of the analysis server.

[0011] FIG. 1 is a diagram for explaining an overview of the present invention. FIG. 2 is a diagram for explaining an overview of the present invention. FIG. 3 is a diagram for explaining an overview of the present invention. FIG. 4 is a diagram for explaining an overview of the present invention. FIG. 5 is a block diagram showing a schematic configuration of a mixed reality system according to an embodiment. FIG. 6 is a sequence diagram showing an example of a processing operation of the mixed reality system.

[0012] An overview of the present invention is shown in Figures 1A to 1F. A detailed description follows. Even if AI (artificial intelligence) is applied to an MR (mixed reality) system, recent AI technologies are only capable of analyzing "text," such as the speed and tone of a user's speech. However, it is difficult to understand a user's emotions from analyzing only "text."

[0013] It is also possible to take a picture of the user with a camera and analyze the user's emotions using AI, but the amount of data captured by the camera is large, which increases the computational load.

[0014] If AI can analyze user emotions, conversations and interactions in MR systems will become more human-like. Therefore, in this embodiment, emotion analysis in an MR system is realized as follows.

[0015] 2 is a block diagram showing a schematic configuration of a mixed reality system according to an embodiment of the present invention. The MR system includes an MR headset, an external camera, and an analysis server.

[0016] The external camera captures the user of the MR headset and the spatial environment and outputs a video, which includes multiple frames.

[0017] The display of the MR headset may be a glasses-type see-through display. The display displays MR visuals and the results of emotion analysis, which will be described later. The displayed MR visuals include a combination of real space and virtual space.

[0018] The MR headset displays the results that are processed and analyzed by the analysis server.

[0019] The analysis server analyzes the user's movements (body posture) and the environment based on data from the external camera, and the results of the analysis are sent to the MR headset.

[0020] 3 is a sequence diagram showing an example of the processing operation of the mixed reality system. First, an external camera captures an image of the user and the environment and acquires video (step S1). This video may be a moving image composed of multiple consecutive frames. Next, the analysis server extracts people from the video data. A specific example of person extraction will be described below.

[0021] The analysis server detects objects from the video (step S2a) and determines whether each detected object is a dynamic object or a static object (step S2b). More specifically, the analysis server makes the determination based on frames. As an example, the analysis server compares the position of each object over five frames. An object whose position in one frame is different from its position in another frame by a predetermined distance or more is determined to be a dynamic object. On the other hand, an object whose position does not move over the five frames is determined to be a static object.

[0022] An external camera may capture static objects in the captured video, but since it is unnecessary to compute frames containing static objects, the overall computation can be reduced, resulting in improved performance.

[0023] The analysis server determines whether each of the dynamic objects is a person (step S2c). A known method can be applied to determine whether each of the dynamic objects is a person.

[0024] The external camera may capture dynamic objects other than the user (e.g., dogs and cats). However, emotion analysis is not required for non-human objects, so computation can be reduced for improved performance.

[0025] Once a person has been extracted in this way, the analysis server records the person's data. The recorded data may be video (or frame) data containing the person, or the person's pattern, which will be described next.

[0026] The analysis server analyzes and records the person's pattern (step S3). The pattern may be a facial pattern or a body pattern. As an example, the analysis server determines whether the person matches a predetermined pattern. The predetermined pattern may be crossed arms, slumped shoulders, clenched fists, etc. Alternatively, the pattern may be a facial pattern (dimples, slanted eyes, etc.).

[0027] The result of the emotion analysis is sent from the analysis server (step S4). For example, if the pattern is "crossing arms," ​​the emotion analyzer 20 analyzes that the user's emotion is defensive. If the pattern is "slumping shoulders," the emotion analyzer 20 analyzes that the user's emotion is depressed. If the pattern is "clenching fists," the emotion analyzer 20 analyzes that the user's emotion is tension, anxiety, or frustration.

[0028] The results of the sentiment analysis are sent from the analysis server to the MR headset, which then displays the results.

[0029] As described above, analyzing user emotions can enhance interaction in MR systems. Furthermore, because emotion analysis is performed on person data (patterns) rather than on the entire video captured by an external camera, processing volume, power consumption, and processing time can be reduced.

[0030] This embodiment can be applied to virtual idols, virtual tour guides, virtual companions (friends, tutors, coaches, etc.), etc. For example, by analyzing the user's emotions, it becomes possible to make the virtual idol react more like a real person, or to have more human-like communication with the virtual companion.

[0031] Any part or all of the functional units described in this specification may be realized by a program. The program mentioned in this specification may be non-transitoryly recorded on a computer-readable recording medium.

[0032] Based on the above description, a person skilled in the art may be able to conceive additional effects and various modifications of the present invention, but the aspects of the present invention are not limited to the individual embodiments described above. For example, inventions that extract only a part of each embodiment or inventions that combine multiple embodiments are naturally envisioned. Various additions, modifications, and partial deletions are possible within the scope of the conceptual idea and spirit of the present invention, which can be derived from the content defined in the claims and their equivalents.

[0033] For example, what is described in this specification as one device (or component, the same applies hereinafter) (including what is depicted as one device in the drawings) may be realized by multiple devices. Conversely, what is described in this specification as multiple devices (including what is depicted as multiple devices in the drawings) may be realized by one device. Alternatively, some or all of the means or functions included in one device may be included in another device. Furthermore, a "system" may be composed of one device, or two or more devices.

[0034] Furthermore, not all of the features described in this specification are essential requirements. In particular, features described in this specification but not included in the claims can be considered optional additional features.

[0035] The problem that the present invention aims to solve should be identified by considering the entire specification. For example, if the specification states that a specific effect is achieved by a specific configuration, it can also be said that the invention solves a problem that is the reverse of the specific effect. However, it is not intended that such a specific configuration is necessarily a required requirement.

Claims

1. A method for emotion analysis in a mixed reality (MR) system, comprising: a first step of capturing an image of a user of an MR headset and a spatial environment with a camera to obtain the image; a second step of extracting a person from the image; a third step of detecting data of the extracted person; a fourth step of analyzing the user's emotions and body posture based on the detected data of the person; and a fifth step of displaying the results of the analysis on the display of the MR headset.

2. The method according to claim 1, wherein in the second step, people are extracted by detecting objects from the video, determining whether each of the detected objects is a dynamic object or a static object, and determining whether each of the dynamic objects is a person.

3. The method of claim 2, wherein the video includes a plurality of frames, and in the second step, it is determined whether each of the detected objects is a dynamic object or a static object by comparing the plurality of frames.

4. The method according to claim 1, wherein in the third step, a person's pattern is detected based on whether the person matches a predetermined pattern, and in the fourth step, a sentiment analysis of the user is performed based on the person's pattern.

5. The method of claim 1, wherein the fourth step includes: analyzing the person's data from the MR headset; analyzing the emotions and body postures by an analysis server; and transmitting the results of the analysis from the analysis server and displaying them on the MR headset.

6. A mixed reality (MR) system comprising: an external camera that captures an image of a user of an MR headset and generates an image; an analysis server that extracts a person from the image, detects data of the extracted person, and analyzes the user's emotions and body posture based on the captured image; and an MR headset display that displays an expression based on the results of the analysis server.

Images