Motion-based user emotion interaction method and system

Abstract

A motion-based user emotion interaction method and system are disclosed. The motion-based user emotion interaction method performed by a computing apparatus, according to an aspect of the present invention, comprises the steps of: acquiring a user image by capturing the movement of a user; if user motion is recognized in the user image, confirming whether or not the recognized user motion is pre-registered emotion-related motion; if the recognized user motion is the emotion-related motion, determining a user emotion corresponding to the user motion; and reflecting the user emotion in a provided service.

Description

Motion-based user emotion interaction method and system

[0001] The present invention relates to a motion-based user emotional interaction method and system.

[0002] Recently, services are expanding beyond virtual reality to include extended reality, such as the metaverse.

[0003] Conventional user interfaces for augmented reality have been extremely limited and restrictive in reflecting user expressions (emotions) by using only verbal input devices such as keyboards and mice.

[0004] Additionally, professional personnel such as game developers and broadcasters use a large amount of expensive equipment to perform gesture recognition and facial expression recognition to recognize user emotions, but this has limitations in applying it to metaverse services for various users.

[0005] In addition, with the recent diversification of video-based services such as online classes and customer consultation services via video chat, there is a growing need to provide customized services to customers by automatically recognizing the other party's emotions through video analysis.

[0006] Accordingly, the present invention has been devised to solve the above-described problems, and provides a motion-based user emotion interaction method and system that recognizes emotions by analyzing user motion in an image.

[0007] In addition, the present invention provides a motion-based user emotion interaction method and system that can increase the recognition rate of emotions by analyzing the connectivity of each motion.

[0008] Other objects of the present invention will become more apparent through the preferred embodiments described below.

[0009] According to one aspect of the present invention, a motion-based user emotion interaction method performed on a computing device is provided, including the steps of: acquiring a user image in which a user's movement is captured; recognizing a user motion in the user image, checking whether the recognized user motion is a pre-registered emotion-related motion, and determining a user emotion corresponding to the user motion if it is an emotion-related motion; and reflecting the user emotion in a provided service. A motion-based user emotion interaction method and a program for executing the method are provided.

[0010] Here, the content of the service provided at that point in time can be utilized to determine the user emotion corresponding to the user motion.

[0011] Additionally, it is possible to determine whether a motion is emotionally related by using the duration after the motion ends for the user motion.

[0012] In addition, if the user motion is an emotion-related motion, it is possible to check whether there is a next consecutive motion within a certain period of time, and if the next consecutive motion is an emotion-unrelated motion, it is possible to determine that the user motion is not an emotion-related motion.

[0013] In addition, if the above-described consecutive next motion is also an emotion-related motion, the user emotion can be determined by checking the correlation with the emotion set for each motion.

[0014] In addition, emotional content including a video or photo that induces multiple emotions is provided to the user, and user emotional information is generated and stored using the user's motion that changes while the emotional content is being played and the content of the emotional content at that point in time, and the user emotion can be determined by further using the stored user emotional information.

[0015] In addition, changes in the user's facial expression can be further confirmed in the user image, and the intensity value of the user's emotion can be calculated based on whether or not the facial expression has changed, and a different method of reflecting the user's emotion to the provided service can be applied based on the intensity value.

[0016]

[0017] According to another aspect of the present invention, a motion-based user emotion interaction system is provided, comprising: a communication unit for acquiring a user image capturing a user's movement; an emotion recognition unit for recognizing a user motion in the user image, and determining whether the recognized user motion is a pre-registered emotion-related motion, and if so, determining a user emotion corresponding to the user motion; and an interaction unit for reflecting the user emotion in a provided service.

[0018]

[0019] Other aspects, features and advantages other than those described above will become apparent from the following drawings, claims and detailed description of the invention.

[0020] According to the present invention, by analyzing user motions in a video and recognizing emotions, it can be utilized in various video-based services such as extended reality, online classes, and video-based customer consultation.

[0021] In addition, according to the present invention, the accuracy of emotion recognition can be improved by analyzing the connectivity of each motion.

[0022] FIG. 1 is a functional block diagram illustrating the configuration of a motion-based user emotion interaction system according to one embodiment of the present invention.

[0023] FIG. 2 is a flowchart schematically illustrating a motion-based user emotional interaction process according to one embodiment of the present invention.

[0024] FIG. 3 is a flowchart illustrating a user emotion recognition process using connectivity of user motions according to one embodiment of the present invention.

[0025] Figure 4 is a table illustrating emotion-related motion information according to one embodiment of the present invention.

[0026] FIG. 5 is a flowchart illustrating a process of applying user emotions using intensity of a user's facial expression according to one embodiment of the present invention.

[0027] The present invention is susceptible to various modifications and embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, but rather to encompass all modifications, equivalents, and alternatives falling within the spirit and technical scope of the present invention.

[0028] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0029] Terms such as "first" and "second" may be used to describe various components, but these components should not be limited by these terms. These terms are used solely to distinguish one component from another. For example, terms such as "first threshold" and "second threshold" described below may be predefined as thresholds that are substantially different or partially identical in value. However, since there is room for confusion when expressed using the same word "threshold," terms such as "first" and "second" will be used together for convenience of distinction.

[0030] The terminology used herein is merely used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this specification, it should be understood that the terms "comprises" or "has" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0031] In addition, it is understood that the components of the embodiments described with reference to each drawing are not limited to the specific embodiments, and may be implemented to be included in other embodiments within the scope in which the technical idea of ​​the present invention is maintained, and that multiple embodiments may be re-implemented as one integrated embodiment even if a separate description is omitted.

[0032] In addition, when describing with reference to the attached drawings, identical components will be assigned identical or related reference numerals regardless of the drawing reference numbers, and redundant descriptions thereof will be omitted. When describing the present invention, if a detailed description of a related known technology is judged to unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

[0033]

[0034] FIG. 1 is a functional block diagram illustrating the configuration of a motion-based user emotion interaction system according to one embodiment of the present invention.

[0035] Referring to FIG. 1, a motion-based user emotion interaction system (hereinafter referred to as the emotion interaction system) according to the present embodiment includes a storage unit (10), a communication unit (20), and a control unit (30), and the control unit (30) may include a user management unit (31), an emotion recognition unit (32), and an interaction unit (33).

[0036] The storage unit (10) stores data necessary for the control unit (30) to function, and also stores user emotion information for customized services. User emotion information will be described in detail later.

[0037] The communication unit (20) is a communication means for providing services utilizing information about user emotions to user terminals connected via a communication network or the like. For example, the communication unit (20) captures video footage of the user's movements from the user terminal and transmits service data reflecting the recognized user emotions to the user terminal. Since such communication means are readily apparent to those skilled in the art, a detailed description thereof will be omitted.

[0038] When a video of a user's movement is acquired from a user terminal, the control unit (30) analyzes the video based on learned emotion recognition technology (using artificial intelligence) to recognize the user's emotion corresponding to the user's motion, and reflects the recognized user emotion in the provided service.

[0039] The emotion recognition unit (32) of the control unit (30) determines the user's emotion by analyzing the captured video by learning learning data on various user motions according to various emotional states. In addition, the emotion recognition unit (32) can increase the accuracy of user emotion recognition through continuous learning while providing a service using artificial intelligence. In particular, the accuracy of user emotion recognition according to motion can be increased by further utilizing user emotion information registered in advance to correspond to the user as described above. The user emotion information is used to provide a user-customized service, and information on user motions according to each emotion is stored in advance as user emotion information and utilized. The user management unit (31) performs a management function of storing, deleting, and updating such user emotion information. A detailed description thereof will be provided later.

[0040] The interaction unit (33) of the control unit (30) reflects user emotions recognized through user motions into the provided service. For example, if providing an avatar service, gestures and facial expressions corresponding to the user's emotions are applied to the user's avatar (see Fig. 7). Of course, this is merely an example, and user emotions can be applied to all video-based services, including augmented reality, online counseling, and online education, in addition to avatar services.

[0041] For reference, eXtended Reality (eXtended Reality) is a term that encompasses mixed reality (MR) technologies, including virtual reality (VR) and augmented reality (AR). While virtual reality (VR) is a technology that allows users to experience a new reality based on 360-degree images, augmented reality (AR) displays information and content through computer graphics (CG) on top of real objects. Although AR and VR are separate technologies, the two technologies are evolving together while complementing each other's shortcomings. However, at this stage, the differences are clear. VR requires a headset-type terminal (HMD) that covers the entire eye, while AR can be expressed with glasses such as Google Glass.

[0042] Extended reality (XR) creates an expanded reality by freely utilizing virtual and augmented reality (VR and AR) technologies individually or in combination. Microsoft's HoloLens, a glasses-style device, can be considered a form of XR because it analyzes real-world space and object information to display optimized 3D holograms. XR is expected to be applied to diverse fields, including education, healthcare, and manufacturing.

[0043] In this extended reality, technology that recognizes the user's real-time emotional state and reflects it in the service is important for interaction with users.

[0044] Additionally, although not shown in the drawing, it is also possible to recognize user emotions corresponding to user motions by utilizing not only video footage of the user, but also a minimal wearable device such as a wristwatch-type device.

[0045]

[0046] FIG. 2 is a flowchart schematically illustrating a motion-based user emotional interaction process according to one embodiment of the present invention.

[0047] Referring to Figure 2, the emotional interaction system acquires a user image capturing the user's movements (S210). The user image can be generated directly by a camera device installed in the system itself, or can be received from a user terminal that captured the user image via a communication network. The user image can be acquired in file form at regular intervals, or can be acquired in real time via streaming. This will be apparent to those skilled in the art, so a detailed description will be omitted.

[0048] User motion is recognized by analyzing the user image (S220). In other words, any motion based on the user's movements is recognized. For example, raising both arms above the head is recognized as a user motion.

[0049] It is checked whether the recognized user motion is a preset emotion-related motion (S230). As will be explained in detail later, referring to Fig. 4, it is checked whether the recognized motion is previously registered as an emotion-related motion.

[0050] In addition, according to an example, whether a user motion is an emotion-related motion can be determined by using the duration after the motion ends. In other words, a duration can be set for each emotion-related motion, and the motion is recognized as an emotion-related motion only if the set duration is satisfied. For example, if a motion of raising a hand to the shoulder is an emotion-related motion and the duration is 2 seconds, even if the user performs the motion of raising a hand to the shoulder, if another motion is performed within 2 seconds, the user motion will not be recognized as an emotion-related motion.

[0051] If the recognized user motion is an emotion-related motion, the corresponding user emotion is determined and reflected in the provided service (S240). For ease of understanding, for example, if the user motion recognized in a video of a student taking an online class is a motion of bringing the right arm to the forehead, the student's user emotion can be recognized as [embarrassment, confusion], and the service can be provided as an AI education service by adding a more detailed explanation of the current class content.

[0052] Here, the content of the service provided at that point in time can be utilized to determine the user emotion corresponding to the user's motion. For example, assuming the current service is online education and the content is math-related and difficult, a student who is unable to solve a problem immediately may be more likely to experience negative emotions such as "worry" or "confusion" due to the difficulty of the problem. Alternatively, a student who immediately solves a problem may be more likely to experience positive emotions such as "pleasure."

[0053]

[0054] To recognize more accurate user emotions, it is necessary to confirm the correlation between consecutive user motions.

[0055] For example, if both arms are raised as Step 1, and both hands are placed on the face as Step 2, Motion A can be recognized as an emotional motion, but if both arms are raised above the head over the face as Step 3, it can be processed as a non-emotional motion.

[0056] FIG. 3 is a flowchart illustrating a user emotion recognition process using connectivity of user motions according to one embodiment of the present invention, and FIG. 4 is a table illustrating emotion-related motion information according to one embodiment of the present invention.

[0057] Referring to FIG. 3, it is determined whether the user motion recognized in the user image is an emotion-related motion (S310 to 320), and if it is an emotion-related motion, it is determined whether the next motion that continues within a certain period of time (e.g., 5 seconds) is an emotion-related motion (S330).

[0058] In other words, if the recognized user motion is an emotion-related motion, it is checked whether there is a next motion within a certain period of time, and whether the next motion is an emotion-related motion or an emotion-unrelated motion.

[0059] If the next consecutive motion is an emotion-free motion, the recognized user motion is processed as a general motion (i.e., non-emotional motion) (S340).

[0060] In contrast, if the next consecutive motion is also an emotion-related motion, the user emotion is determined by checking the correlation with the emotion set for each motion (S350).

[0061] Referring to Fig. 4, which illustrates information about preset and stored emotion-related motions, the user emotions recognized for each connected motion may be set differently as emotion-related motion information. In other words, even for the emotion-related motion of raising both hands to the face, the user emotions may be recognized differently depending on which connected motion is performed thereafter.

[0062]

[0063] As described above, according to an example, user emotion information can be utilized to increase the accuracy of emotion recognition tailored to each user.

[0064] To achieve this, emotional content, including videos or photos that trigger multiple negative emotions, is provided to users. Footage of the user while the emotional content is playing is acquired and analyzed. Then, changes in user motion are determined. If changes occur, the current emotional content and the user's motion at that time are generated and stored as user emotion information.

[0065] For example, if humorous content is featured in emotional content and the user performs a specific motion, the video feature information and information about the emotional state of laughter associated with that motion can be stored as user emotion information. Of course, even in this case, a recognition process utilizing learning data can be added to determine the emotional state of the user's motion.

[0066] Additionally, user emotion information can be generated based on learning data corresponding to the user's age, gender, and facial shape. For example, since the behavior of a teenager may differ from that of a person in their 40s, learning data corresponding to the user's age, gender, and body type can be prioritized for analyzing user motion to generate more useful user emotion information.

[0067] According to this embodiment, by providing users with content that induces various emotional states in advance and having them watch it, observing user behavior at that time, specifying the characteristics of user motions in each emotional state, and utilizing this for later emotion recognition, the accuracy of emotion recognition can be increased.

[0068]

[0069] FIG. 5 is a flowchart illustrating a process of applying user emotions using intensity of a user's facial expression according to one embodiment of the present invention.

[0070] Referring to Fig. 5, when a user emotion is recognized according to a user motion, the change in the user's facial expression is further confirmed in the user image in response to the recognition time of the user motion (S510).

[0071] Then, the intensity value of the user's emotion is calculated based on changes in facial expression (S520). Of course, the greater the change in facial expression, the higher the intensity value can be calculated.

[0072] Depending on the intensity value, the method of reflecting user emotions in the provided service varies. For example, if the intensity value is high, the avatar's actions corresponding to the emotional state [laughter] are expressed in a larger scale and special effects are inserted. Conversely, if the intensity value is low, only small gestures related to [laughter] are reflected on the avatar without any special effects.

[0073]

[0074] A computer program stored in a computer-readable medium that performs the motion-based user emotional interaction method according to the present invention described above may be provided.

[0075] Additionally, the motion-based user emotional interaction method described above can be implemented as computer-readable code on a computer-readable recording medium. Computer-readable recording media include all types of recording media that store data decipherable by a computer system. Examples include read-only memory (ROM), random access memory (RAM), magnetic tape, magnetic disks, flash memory, and optical data storage devices. Furthermore, the computer-readable recording medium can be distributed across computer systems connected via a computer communications network, and stored and executed as readable code in a distributed manner.

[0076] In addition, although the above has been described with reference to preferred embodiments of the present invention, it will be understood by those skilled in the art that various modifications and changes can be made to the present invention without departing from the spirit and scope of the present invention as set forth in the claims below.

Claims

1. In a method of user emotional interaction performed on a computing device, A step of acquiring a user image that captures the user's movements; When a user motion is recognized in the above user image, a step of checking whether the recognized user motion is a pre-registered emotion-related motion, and if it is an emotion-related motion, determining a user emotion corresponding to the user motion; and A motion-based user emotion interaction method, comprising a step of reflecting the user emotion in a provided service.

2. In claim 1, A motion-based user emotion interaction method that utilizes the content of the service provided at the time to determine the user emotion corresponding to the user motion.

3. In claim 1, A motion-based user emotion interaction method that determines whether a motion is emotion-related by using the duration after the end of the motion for the user motion.

4. In claim 1, A motion-based user emotion interaction method, wherein if the above user motion is an emotion-related motion, it is checked whether a subsequent motion exists within a certain period of time, and if the subsequent motion is an emotion-unrelated motion, it is determined that the above user motion is not an emotion-related motion.

5. In claim 4, A motion-based user emotion interaction method that determines the user emotion by checking the correlation for the emotion set for each motion when the above consecutive next motion is also an emotion-related motion.

6. In claim 1, A motion-based user emotion interaction method, which provides a user with emotion-inducing content including a video or photo that induces multiple emotions, generates and stores user emotion information using the user's motion that changes while the emotion-inducing content is being played and the content of the emotion-inducing content at that point in time, and further uses the stored user emotion information to determine the user emotion.

7. In claim 1, In the above user video, the change in the user's facial expression is further confirmed, and the intensity value of the user's emotion is calculated based on whether or not the facial expression has changed. A motion-based user emotion interaction method that applies different reflection methods for the user emotion to the provided service according to the above intensity value.

8. A computer program stored in a computer-readable medium that causes a computer to perform a motion-based user emotion interaction method, wherein the computer program causes a computer to perform the following steps, wherein the steps are: A step of acquiring a user image that captures the user's movements; When a user motion is recognized in the above user image, a step of checking whether the recognized user motion is a pre-registered emotion-related motion, and if it is an emotion-related motion, determining a user emotion corresponding to the user motion; and A computer program stored in a computer-readable medium, comprising a step of reflecting the user's emotions in the provided service.

9. A communication unit for acquiring user images that capture the user's movements; An emotion recognition unit that, when recognizing a user motion in the above user image, checks whether the recognized user motion is a pre-registered emotion-related motion, and if it is an emotion-related motion, determines the user emotion corresponding to the user motion; and A motion-based user emotion interaction system including an interaction unit that reflects the user's emotions in the provided service.

Images