Method and apparatus for emotion recognizing based on context information

Abstract

A context information-based emotion recognition method according to one embodiment of the present invention comprises: a step of detecting information corresponding to an emotion recognition target from an input image; a step of extracting recognition target features based on the information corresponding to the emotion recognition target; a step of extracting context features based on the input image; a step of storing the recognition target features and the context features in a short-term memory; and a step of storing the context features in a long-term memory.

Description

Technology Field

[0001] The present invention relates to a technology for estimating human emotions by analyzing input images.

[0002] Specifically, the present invention relates to a technology for estimating human emotions within an input image based on an artificial intelligence neural network.

[0003] Specifically, the present invention relates to a technology for estimating a person's emotions within an image using contextual information within an input image. Background Technology

[0004] The video-based automated emotion recognition method observes and analyzes the user's voluntary external expressions to analyze and infer the user's emotions, which are an invisible internal state.

[0005] Conventional automated emotion recognition methods primarily focus on identifying emotions by analyzing user video and audio data to extract facial expressions, behaviors, tone of voice, and linguistic features, and then training deep neural networks (DNNs) with these features alongside related emotion annotation information to accurately determine a person's internal emotional state. However, to accurately understand human emotions, not only the user's external outward characteristics (facial expressions, behaviors, tone of voice, and language) but also the temporal and spatial contextual features related to the user must be considered. For example, periodic contextual information may be necessary because people's facial expressions may be somber upon returning to an office on Monday after a weekend break. The emotional state of patients in a hospital may exhibit characteristics different from their emotional state in other indoor environments. Generally, people tend to identify emotions by considering these spatiotemporal contexts when judging the emotions of others.

[0006] The present invention relates to a context-aware emotion recognition method based on long-term and short-term memory for accurately understanding a person's complex internal emotional state. To consider temporal context, long-term and short-term emotional memory features are stored in memory, and emotion recognition is performed by conducting an integrated analysis together with the user's emotional feature information when necessary. Prior art literature

[0007] Korean Registered Patent Publication No. 2398683 (Title of Invention: Construction of an Emotion Dictionary Using Paraphrasing and System and Method for Recognizing Emotion Structure in Text Using the Same) The problem to be solved

[0008] The objective of the present invention is to provide more accurate emotion recognition results through context-aware emotion recognition.

[0009] In addition, the objective of the present invention is to efficiently store context information by utilizing short-term and long-term memory and to perform emotion recognition using the stored information. means of solving the problem

[0010] A context information-based emotion recognition method according to an embodiment of the present invention for achieving the above-mentioned purpose comprises: a step of detecting information corresponding to an emotion recognition target from an input image; a step of extracting recognition target features based on the information corresponding to the emotion recognition target; a step of extracting context features based on the input image; a step of storing the recognition target features and the context features in a short-term memory; and a step of storing the context features in a long-term memory.

[0011] At this time, the long-term memory can store context text information generated based on the context features.

[0012] At this time, the method may further include a step of recognizing the emotion of the emotion recognition target based on information stored in the short-term memory and the long-term memory.

[0013] At this time, the step of recognizing the emotion can recognize the emotion by using summary context information generated from the context text information stored in the long-term memory as input.

[0014] At this time, the above contextual text information can be generated by inputting the above contextual features into a pre-trained language model.

[0015] At this time, the information corresponding to the emotion recognition target may include the face area of ​​the emotion recognition target, the full body area of ​​the emotion recognition target, and the voice of the emotion recognition target.

[0016] At this time, the short-term memory can store the recognition target feature and the context feature at a first frequency for the entire frame of the input image.

[0017] At this time, the long-term memory can store the context text information at a second frequency lower than the first frequency for all frames of the input image.

[0019] Additionally, a context information-based emotion recognition device according to an embodiment of the present invention for achieving the above-mentioned purpose comprises: a target detection unit that detects information corresponding to an emotion recognition target from an input image; a target feature extraction unit that extracts recognition target features based on the information corresponding to the emotion recognition target; a context feature extraction unit that extracts context features based on the input image; a short-term memory storage unit that stores the recognition target features and the context features; and a long-term memory storage unit that stores the context features.

[0020] At this time, the long-term memory storage unit can store context text information generated based on the context features.

[0021] At this time, the device may further include an emotion recognition unit that recognizes the emotion of the emotion recognition target based on information stored in the short-term memory storage unit and the long-term memory storage unit.

[0022] At this time, the emotion recognition unit can recognize emotions using summary context information generated from the context text information stored in the long-term memory storage unit as input.

[0023] At this time, the above contextual text information can be generated by inputting the above contextual features into a pre-trained language model.

[0024] At this time, the information corresponding to the emotion recognition target may include the face area of ​​the emotion recognition target, the full body area of ​​the emotion recognition target, and the voice of the emotion recognition target.

[0025] At this time, the short-term memory storage unit can store the recognition target feature and the context feature at a first frequency for all frames of the input image.

[0026] At this time, the long-term memory storage unit can store the context text information at a second frequency lower than the first frequency for all frames of the input image. Effects of the invention

[0027] According to the present invention, more accurate emotion recognition results can be provided through context-aware emotion recognition.

[0028] In addition, the present invention can efficiently store context information by utilizing short-term and long-term memory and perform emotion recognition using the stored information. Brief explanation of the drawing

[0029] FIG. 1 is a flowchart illustrating a context information-based emotion recognition method according to one embodiment of the present invention. FIG. 2 is a configuration diagram of a context information-based emotion recognition system according to one embodiment of the present invention. Figure 3 is an example of a pre-processor configuration. Figure 4 shows an example of learning an emotion model (EM). Figure 5 shows an example of training a vision model (VM) and a language model (LM). Figure 6 illustrates a method for training a language model (LM) through a document summarization task. Figure 7 shows an example of learning an emotion recognizer (ER). FIG. 8 is a block diagram showing a context information-based emotion recognition device according to one embodiment of the present invention. FIG. 9 is a diagram showing the configuration of a computer system according to an embodiment. Specific details for implementing the invention

[0030] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Throughout the specification, the same reference numerals refer to the same components.

[0031] Although terms such as "first" or "second" are used to describe various components, these components are not limited by such terms. Such terms may be used merely to distinguish one component from another. Accordingly, the first component mentioned below may be the second component within the technical scope of the present invention.

[0032] The terms used herein are for describing embodiments and are not intended to limit the invention. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text. As used in this specification, "comprises" or "comprising" implies that the mentioned component or step does not exclude the presence or addition of one or more other components or steps.

[0033] In this specification, each of the phrases such as “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “at least one of A, B, or C” may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof.

[0034] Unless otherwise defined, all terms used in this specification may be interpreted in a sense that is commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise.

[0035] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. When describing with reference to the drawings, identical or corresponding components are given the same reference numerals, and redundant descriptions thereof will be omitted.

[0036] The present invention relates to a system that integrates and infers a user's emotions based on spatiotemporal context information and information such as the user's facial expressions and behaviors, utilizing long-term and short-term memory for the analysis and inference of user emotions.

[0037] The present invention can operate on camera equipment, such as a CCTV capable of capturing real-time RGB images, and on a computer capable of analyzing real-time images from said equipment. The computer may include volatile memory, such as RAM (Random Access Memory), for storing short-term emotional information while a program is running, and non-volatile storage, such as a hard disk, for storing long-term emotional information.

[0038] The present invention can be utilized in applications requiring long-term emotional state observation, such as long-term emotional monitoring of the elderly, emotional monitoring of long-term patients, emotional monitoring of VIP customers at department stores, and emotional monitoring systems for membership customers of restaurant serving robots.

[0039] At this time, the present invention is characterized by individual configurations of long-term and short-term memory for considering temporal context, configurations regarding the utilization of a Vision Model (VM) and a Language Model (LM) for the effective combination of spatiotemporal emotional context features, and configurations regarding the utilization of an LM for efficient text storage / log verification / summarization (compression) / extraction of long-term emotional context features.

[0040] FIG. 1 is a flowchart illustrating a context information-based emotion recognition method according to one embodiment of the present invention.

[0041] A context information-based emotion recognition method according to one embodiment of the present invention can be performed on an emotion recognition device such as a computing device.

[0042] Referring to FIG. 1, a context information-based emotion recognition method according to one embodiment of the present invention includes the steps of: detecting information corresponding to an emotion recognition target from an input image (S110); extracting recognition target features based on the information corresponding to the emotion recognition target (S120); extracting context features based on the input image (S130); storing the recognition target features and the context features in a short-term memory (S140); and storing the context features in a long-term memory (S150).

[0043] At this time, the long-term memory can store context text information generated based on the context features.

[0044] At this time, although not illustrated in FIG. 1, the method may further include the step of recognizing the emotion of the emotion recognition target based on information stored in the short-term memory and the long-term memory.

[0045] At this time, the step of recognizing the emotion can recognize the emotion by using summary context information generated from the context text information stored in the long-term memory as input.

[0046] At this time, the above contextual text information can be generated by inputting the above contextual features into a pre-trained language model.

[0047] At this time, the information corresponding to the emotion recognition target may include the face area of ​​the emotion recognition target, the full body area of ​​the emotion recognition target, and the voice of the emotion recognition target.

[0048] At this time, the short-term memory can store the recognition target feature and the context feature at a first frequency for the entire frame of the input image.

[0049] At this time, the long-term memory can store the context text information at a second frequency lower than the first frequency for all frames of the input image.

[0050] FIG. 2 is a configuration diagram of a context information-based emotion recognition system according to one embodiment of the present invention.

[0051] VM, EM, LM, LTM, STM, and ER described in FIG. 2 may represent a vision model, an emotion model, a language model, a long-term memory, a short-term memory, and an emotion recognizer, respectively. Additionally, each module may correspond to an artificial intelligence neural network, but the scope of the present invention is not limited thereto.

[0052] Referring to FIG. 2, a context information-based emotion recognition system (device) according to one embodiment of the present invention may receive one frame of a real-time captured video and a face image (still image) of a target to be analyzed. The real-time captured video is input into the system frame by frame, and the user's face image is used to find the target to be emotion analyzed within the real-time captured video.

[0053] Figure 3 is an example of a pre-processor configuration.

[0054] The preprocessor of FIG. 2 is intended for preprocessing to utilize input information. Referring to FIG. 3, in a method according to one embodiment of the present invention, the preprocessor can detect the user's face region and full body region through human detection, face detection, and identity recognition within an image. In another embodiment, the preprocessor can extract only the user's voice information by separating and recognizing the speaker to make the user's voice information usable. Additionally, the preprocessor can perform text information extraction to extract meaning, and can also perform modality-based input preprocessing.

[0055] In this case, the Emotion Model (EM) is a model that extracts emotional features of the user's appearance based on the results received from the preprocessor. The Emotion Model (EM) is a multi-task model that performs the role of extracting emotional features from various inputs.

[0056] In one embodiment, the emotion model (EM) can receive facial expression images and full-body behavior images as input and convert them into emotion features. In another embodiment, the emotion model (EM) can also convert the user's voice / language information into emotion features. Facial expressions, full-body behavior, voice, language, etc., can be used in combination, and other information capable of directly representing the user's emotions can also be input.

[0057] Figure 4 shows an example of learning an emotion model (EM).

[0058] Referring to Fig. 4, when training an emotion model with facial expression images and whole-body behavior images, a Fully-Connected (FC) Layer is attached for result generation, and training can be performed based on input image and emotion annotation pairs; after training, the FC can be removed and utilized. The emotion model (EM) can be trained in the same way when using other modalities (speech, language). The emotion model (EM) may also use a feature extractor that has learned emotion results from various input modalities without separate training.

[0059] The Vision Model (VM) receives a single frame captured in real time and performs the role of extracting features from the entire image. In other words, the Vision Model (VM) performs the role of extracting spatial context features that represent spatial information, the relationship between the user and others within the image, and the situation. The Language Model (LM) performs the role of expressing image features as text and describes the image in text by utilizing the image features received from the Vision Model (VM).

[0060] Figure 5 shows an example of training a vision model (VM) and a language model (LM).

[0061] Referring to Fig. 5, a vision model (VM) and a language model (LM) can be trained based on tasks such as image captioning as shown in Fig. 5. Features representing contextual information within an image can be extracted from the trained vision model (VM), and text describing the image can be extracted through the language model (LM). The vision model (VM) and the language model (LM) can utilize a pre-trained image captioning model.

[0062] Short-term Memory (STM) stores context features (image features in Fig. 5) extracted from the Vision Model (VM) and emotion features from the Emotion Model (EM). Based on user parameters, the STM can specify the short-term memory period (number of frames, len) and the skip period (skip_len) to prevent excessive redundancy. That is, when the current frame is i, skip_len is 1, and len is 4, the context features and emotion features of the i-3rd, i-2nd, i-1st, and i-th frames are stored in the STM. The STM stores data in volatile memory such as RAM. In other words, while the program is running, frame-specific emotion features are stored based on user parameters (skip_len, len, etc.), and they are initialized upon termination.

[0063] Unlike short-term memory (STM), long-term memory (LTM) stores sentiment-described texts extracted from a vision model-language model (VM-LM) on non-volatile storage (such as HDD). To store context features over the long term on a daily, monthly, or yearly basis, the storage capacity of the computer on which the program runs must be considered. To appropriately represent contextual information present in images and optimize storage space, sentences describing the image context are stored frame by frame as sentences or paragraphs. Storing images as descriptive sentences offers the following advantages.

[0064] Since images are represented as text, frame-by-frame emotional context information in daily, monthly, and yearly units can be expressed as semantic elements, thereby achieving the effect of compressing storage space. Furthermore, because language is utilized similarly to the emotional context information in human long-term memory, long-term emotional memories can be appropriately represented. Additionally, since context information is stored as text, it is easy to check the user's emotional information logs. Moreover, when retrieving sentences from long-term emotional memories to utilize temporal context information, a language model (LM) can be employed to extract summary features composed of important information.

[0065] Similar to short-term memory (STM), long-term memory (LTM) also allows determining the density and amount of contextual information to be stored by specifying the long-term storage period (len) and skip period (skip_len) based on user parameters. For example, if skip_len=20, len=51, and current frame=i, the image description sentences for frames i-1000, i-980, …, i-40, i-20, and i will be stored in the LTM. Text information storing temporal context frame by frame is extracted from long-term memory (LTM) when emotion recognition is performed. At this stage, rather than utilizing all long-term text information, the characteristic of summarizing only key information—similar to that of humans—is utilized. This is achieved through the document summarization task of the language model (LM).

[0066] Figure 6 illustrates a method for training a language model (LM) through a document summarization task.

[0067] The language model (LM) may also utilize a pre-trained model that has been trained on a document summarization task. The language model (LM) must be capable of processing the task of expressing image features received from the vision model (VM) as text and the document summarization task. As one embodiment of the language model (LM), a DNN with two heads can be utilized to enable two output results through multi-task learning.

[0068] Long-term text information extracted from long-term memory (LTM) is summarized into long-term summary features through a language model (LM). The summarized long-term summary features are input to the final stage, the Emotion Recognizer (ER), along with short-term features from short-term memory (STM), and the Emotion Recognizer (ER) infers the final emotional state based on the input features.

[0070] Figure 7 shows an example of learning an emotion recognizer (ER).

[0071] Referring to Fig. 7, to train the emotion recognizer (ER), all other models can be frozen (locked so they are not trained), and user parameters of the long / short-term memory (LTM / STM) can be fixed and training can be performed.

[0072] Additionally, as an example of training, other models excluding the Emotion Recognizer (ER) may be pre-trained, and after the user sets the parameters, the Emotion Recognizer (ER) may be trained by running the program in training mode for a certain period while collecting data and emotion annotations. In another example, data and emotion annotations may all be collected in offline mode and then trained all at once.

[0073] The preprocessor, vision model (VM), language model (LM), sentiment model (EM), and sentiment recognizer (ER) described above can be trained and used through DNN, but other types of trainable algorithms can also be utilized. Long-term memory (LTM) can use non-volatile storage capable of storing data even after the program terminates. Short-term memory (STM) may use volatile storage such as RAM, but it can also be used as non-volatile storage. In this case, log records in short-term memory (STM) can also be checked even after the program terminates.

[0074] FIG. 8 is a block diagram showing a context information-based emotion recognition device according to one embodiment of the present invention.

[0075] Referring to FIG. 8, a context information-based emotion recognition device according to one embodiment of the present invention includes a target detection unit (810) that detects information corresponding to an emotion recognition target from an input image, a target feature extraction unit (820) that extracts recognition target features based on the information corresponding to the emotion recognition target, a context feature extraction unit (830) that extracts context features based on the input image, a short-term memory storage unit (840) that stores the recognition target features and the context features, and a long-term memory storage unit (850) that stores the context features.

[0076] At this time, the long-term memory storage unit (850) can store context text information generated based on the context features.

[0077] At this time, the device may further include an emotion recognition unit that recognizes the emotion of the emotion recognition target based on information stored in the short-term memory storage unit and the long-term memory storage unit.

[0078] At this time, the emotion recognition unit can recognize emotions using summary context information generated from the context text information stored in the long-term memory storage unit as input.

[0079] At this time, the above contextual text information can be generated by inputting the above contextual features into a pre-trained language model.

[0080] At this time, the information corresponding to the emotion recognition target may include the face area of ​​the emotion recognition target, the full body area of ​​the emotion recognition target, and the voice of the emotion recognition target.

[0081] At this time, the short-term memory storage unit (840) can store the recognition target feature and the context feature at a first frequency for all frames of the input image.

[0082] At this time, the long-term memory storage unit (850) can store the context text information at a second frequency lower than the first frequency for all frames of the input image.

[0083] FIG. 9 is a diagram showing the configuration of a computer system according to an embodiment.

[0084] A context information-based emotion recognition device according to an embodiment can be implemented in a computer system (1000), such as a computer-readable recording medium.

[0085] A computer system (1000) may include one or more processors (1010), memory (1030), user interface input device (1040), user interface output device (1050), and storage (1060) that communicate with each other via a bus (1020). Additionally, the computer system (1000) may further include a network interface (1070) connected to a network (1080). The processor (1010) may be a semiconductor device that executes programs or processing instructions stored in a central processing unit, memory (1030), or storage (1060). The memory (1030) and storage (1060) may be storage media comprising at least one of a volatile medium, a non-volatile medium, a removable medium, a non-removable medium, a communication medium, or an information transfer medium. For example, the memory (1030) may include a ROM (1031) or a RAM (1032).

[0086] The specific embodiments described in this invention are examples and do not limit the scope of the invention in any way. For the sake of brevity of the specification, descriptions of prior electronic configurations, control systems, software, and other functional aspects of said systems may be omitted. Additionally, the connections of lines or connecting members between components shown in the drawings are illustrative of functional connections and / or physical or circuit connections, and may be replaced or additionally represented as various functional connections, physical connections, or circuit connections in actual devices. Furthermore, unless specifically stated as “essential,” “importantly,” etc., a component may not be strictly necessary for the application of the invention.

[0087] Accordingly, the scope of the present invention should not be limited to the embodiments described above, and all scopes equivalent to or equivalently modified from the claims set forth below, as well as the claims set forth below, shall be considered to fall within the scope of the concept of the present invention. Explanation of the symbols

[0088] 1000: Computer System 1010: Processor 1020: Bus 1030: Memory 1031: ROM 1032: RAM 1040: User interface input device 1050: User interface output device 1060: Storage 1070: Network Interface 1080: Network

Claims

Claim 1 A context-aware emotion recognition method based on short-term and long-term memory performed in an emotion recognition device, comprising: a step of detecting information corresponding to an emotion recognition target from an input image; a step of extracting recognition target features based on the information corresponding to the emotion recognition target; a step of extracting context features based on the input image; a step of storing the recognition target features and the context features in a short-term memory; and a step of storing the context features in a long-term memory; wherein the context features are extracted based on a frame of the input image and include relationship information or situation information within the input image. Claim 2 A context information-based emotion recognition method according to claim 1, wherein the long-term memory stores context text information generated based on the context features. Claim 3 A context information-based emotion recognition method according to claim 2, further comprising the step of recognizing the emotion of an emotion recognition target based on information stored in the short-term memory and the long-term memory. Claim 4 A context information-based emotion recognition method according to claim 3, wherein the step of recognizing the emotion is characterized by recognizing the emotion using summary context information generated from context text information stored in the long-term memory as input. Claim 5 A context information-based emotion recognition method according to claim 2, characterized in that the context text information is generated by inputting the context features into a pre-trained language model. Claim 6 A context information-based emotion recognition method according to claim 1, characterized in that the information corresponding to the emotion recognition target includes a face region of the emotion recognition target, a whole body region of the emotion recognition target, and the voice of the emotion recognition target. Claim 7 A context information-based emotion recognition method according to claim 2, wherein the short-term memory stores the recognition target feature and the context feature at a first frequency for the entire frame of the input image. Claim 8 A context information-based emotion recognition method according to claim 7, wherein the long-term memory stores the context text information at a second frequency lower than the first frequency for the entire frame of the input image. Claim 9 A context information-based emotion recognition device comprising: a target detection unit that detects information corresponding to an emotion recognition target from an input image; a target feature extraction unit that extracts recognition target features based on the information corresponding to the emotion recognition target; a context feature extraction unit that extracts context features based on the input image; a short-term memory storage unit that stores the recognition target features and the context features; and a long-term memory storage unit that stores the context features; wherein the context features are extracted based on the frames of the input image and include relationship information or situation information within the input image. Claim 10 A context information-based emotion recognition device according to claim 9, wherein the long-term memory storage unit stores context text information generated based on the context features. Claim 11 A context information-based emotion recognition device according to claim 10, wherein the device further comprises an emotion recognition unit that recognizes the emotion of the emotion recognition target based on information stored in the short-term memory storage unit and the long-term memory storage unit. Claim 12 A context information-based emotion recognition device according to claim 11, wherein the emotion recognition unit recognizes an emotion using summary context information generated from the context text information stored in the long-term memory storage unit as input. Claim 13 A context information-based emotion recognition device according to claim 10, characterized in that the context text information is generated by inputting the context features into a pre-trained language model. Claim 14 A context information-based emotion recognition device according to claim 9, characterized in that the information corresponding to the emotion recognition target includes the face region of the emotion recognition target, the whole body region of the emotion recognition target, and the voice of the emotion recognition target. Claim 15 A context information-based emotion recognition device according to claim 10, wherein the short-term memory storage unit stores the recognition target feature and the context feature at a first frequency for the entire frame of the input image. Claim 16 A context information-based emotion recognition device according to claim 15, wherein the long-term memory storage unit stores the context text information at a second frequency lower than the first frequency for all frames of the input image.

Images