A deep reinforcement learning framework for representing video content
Through the deep reinforcement learning framework interaction with environmental state, combined with convolutional neural network and reward mechanism, the automation problem of emotional label prediction in movie scenes is solved, efficient emotion classification under weak supervision conditions is achieved, and manual labeling costs are reduced and increased accuracy.
Patent Information
- Application Number
- CN201880069825.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2017-10-27
- Filing Date
- 2018-10-25
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2038-10-25
AI Technical Summary
The prior art is difficult to effectively automate the prediction of emotional labels in movie scenes, especially in the absence of sufficient supervised labeling data and complex multimodal information processing, resulting in complex and costly emotional classification tasks.
Using the deep reinforcement learning (DRL) framework, by modifying the traditional DRL structure, the agent can interact with the environmental state of the video scene, use a pre-trained convolutional neural network to capture emotional information from facial expressions, and optimize emotional labels through reward mechanisms Prediction, generate training sequences and facial embeddings to improve prediction accuracy.
It realizes efficient prediction of emotional labels for movie scenes under weak supervision, reduces manual labeling costs, improves the accuracy of emotional classification and data utilization, and can generate output emotional labels in real time.
Smart Images

Figure CN111837142B_ABST
Abstract
Description
[0001] Priority claim
[0002] This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 577,970, filed on October 27, 2017, the entire contents of which are incorporated herein by reference. Background Art
[0003] With the popularity of video-on-demand mobile phone applications and websites such as youtube.com, people have access to an unprecedented amount of videos from different online sources. Therefore, extracting video metadata (e.g., movie scene annotations) plays an important role in facilitating efficient search and recommendation. For example, a movie recommendation system needs to identify the emotional information of movie clip scenes and recommend them to the corresponding users based on their requests or interests.
[0004] However, most reliable movie annotations are still manually generated because movie scene emotion labels, such as whether a scene is funny or not, are usually manifested on a longer time scale. When human annotators try to identify the emotion labels of movie scenes, they need to consider a longer context. It is not easy to simulate this complex nonlinear nature of the human annotation process using a specific algorithm. In addition, movie emotion label prediction is a more challenging and complex task than emotion prediction using a well-prepared standard emotion database. In addition, scene-level emotion labels are usually generated from multiple modalities. For example, movies contain information from visual, auditory, and lexical channels. When an annotator labels a movie scene as funny, the evaluation may be because of the actors' facial expressions, the transcripts they use, or just the background music. More generally, annotators usually combine all this information to evaluate whether a scene is funny or not. Although a large number of movie audio resources are available, those with accurate scene-level annotations are usually severely limited in number due to copyright and annotation costs. Therefore, emotion classification of movie scenes is a complex and challenging task both in terms of technical algorithms and data collection.
[0005] Many related research works have focused on the analysis of emotional content in movies. Recently, deep learning methods such as convolutional neural networks (CNNs) have been used to extract mid-level representations and applied them to emotional video understanding. Most of the related research works in this field have used supervised machine learning methods.
[0006] It is against this background that aspects of the present disclosure arise. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] Figure 1 is a schematic diagram illustrating a proposed RL framework for movie emotion label prediction according to aspects of the present disclosure.
[0008] Figure 2 is a schematic diagram illustrating a proposed movie sequence generation method according to aspects of the present disclosure.
[0009] Figure 3 is a flow chart illustrating a method for performing sequence level prediction of video scenes according to aspects of the present disclosure.
[0010] Figure 4 is a block diagram illustrating a system for performing sequence level prediction of video scenes in accordance with aspects of the present disclosure. DETAILED DESCRIPTION
[0011] Introduction
[0012] At a high level, the concept behind the present disclosure is to use artificial intelligence (AI) methods to improve systems for annotating video content with emotional labels through a trial-and-error mechanism. Reinforcement learning (RL) is a framework for experience-driven autonomous learning methods. With the application of deep learning in RL (defined fields), "deep reinforcement learning" (DRL) is playing a revolutionary role in the field of machine learning. The rise of DRL comes mainly from two aspects. The first aspect is the powerful function approximation and representation learning properties of deep neural networks (DNNs). Using DNNs, low-dimensional feature representations can be obtained from high-dimensional data (e.g., images, audio, and text), which allows DRL to solve decision-making problems with high-dimensional state and action spaces. The second aspect is the "exploration and development" characteristics of RL. Exploration is related to collecting more information, which means that the system will explore different possible trials to see if they are better than those previously tried. Exploration ensures that the system makes the best decision given the current information, which means that the system remembers the most effective strategies in the past. These two advantages are significant improvements over many other traditional supervised learning methods.
[0013] Aspects of the present disclosure are directed to a novel approach that uses deep reinforcement learning (DRL) to perform sequence-level predictions, for example, in video content (e.g., movies, television shows, etc.). Supervision in the form of ground truth labels is only provided at the end of the sequence, and the labels themselves belong to the entire sequence. This problem can be formulated as a game in which a DRL agent attempts to "guess" a global label given only data up to the current time step. This "guess" represents the agent's intrinsic belief about the label and influences any further decisions made by the agent. This belief is also used to calculate a reward function that helps train the DRL algorithm. Different strategies for calculating the reward function have been explored, and it has been shown that weakly supervised methods such as DRL may be useful in tasks such as emotion prediction, where labeling at the global level may be easier. An implementation according to one aspect of the present disclosure includes an algorithm for implementing movie scene emotion label prediction. In this example, the algorithm uses a pre-trained convolutional network to capture complex human emotional information from the faces of the actors in the scene to understand whether a particular movie scene is funny. The traditional DRL framework cannot be directly applied to this task. In traditional DRL, the decisions generated for each frame must interact with the environment to change its state, for example, in a Pong game or AlphaGO. Since the movie scene data sequence is already determined, the predicted emotion label decision for each frame cannot interact with the environment. The slightly modified DRL framework implemented by this algorithm makes it possible for the DRL agent to interact with the state of the environment. This will be described in detail below.
[0014] Methodology
[0015] Movie scene clips can be viewed as sequences of data samples along different modalities. For example, from the speech modality, a clip can contain an audio signal, and from the visual modality, there can be a sequence of image frames in each movie scene clip. For the emotion label prediction task, human annotators need to process the dynamic temporal emotion information to obtain the correct label. For RL applications, the agent's action decision also contains strong temporal correlation, and the current action decision plus the reward also depends on the previous step. Therefore, we try to use RL to predict the emotion label of a movie clip.
[0016] By way of example and not limitation, the focus may be restricted to video data using as input a sequence of images from a set of video frames.The modified DRL algorithm attempts to predict an emotional scene label based on an analysis of one or more actors' faces in each movie frame.
[0017] Framework of the proposed method
[0018] According to aspects of the present disclosure, a new DRL framework includes interactions between environment states and agent actions. Figure 1The proposed RL framework is schematically shown in Figure .
[0019] Video information in a movie scene can be represented as a sequence of faces depicted in each frame. At each time step t, the state of the environment has two components, the video frame at time step t and the predicted emotional information from the previous time step t-1. By jointly considering this information, the agent controlled by the machine learning algorithm takes an action A(t). The output of the action represents the prediction of the emotional label at time step t, such as funny or not funny. The function G is then used to transform the pool of predicted actions {A(i):i≤t} so far into a predicted emotional history for the next time step t+1. This information is included as part of the state of the environment at the next time step and is similarly used by the agent to generate the next action at time t+1. At each step, a reward is generated based on the predicted action by comparing the predicted action up to the current time step t with the corresponding human-annotated movie scene emotional label.
[0020] In this proposed RL structure, we use a function G to add the prediction from the agent's action at the previous time step to the current environment state, thereby establishing a connection between the agent's action and the environment state. Using the proposed RL structure, different functions G and H can be designed for different applications.
[0021] Video data processing
[0022] Emotion label classification datasets can be created from raw video data (e.g., original movies) and corresponding annotation files. For example, scenes in a movie can be labeled as funny or not or one can focus on specific features depicted in the video channel of the movie, especially the faces of the characters. Sequences of these facial images are pre-extracted from the video and used as input to our system. To detect the face of each frame, standard face detection libraries such as dlib and openCV can be used to extract facial images from the raw video of each consecutive frame. If a frame contains multiple faces, the face closest to the center of the frame can be selected. The intuition behind this selection is that when multiple faces are shown on the screen, the face of the main character is likely to be located in the center of the screen to dominate the emotional information of the scene. In addition, the faces selected by this criterion are usually also the largest in area compared to other detected faces.
[0023] If there are multiple faces in a frame, the most centrally located face may be selected, since the main character is likely to be located in the center of the screen to dominate the emotional information of the scene, and typically this selected face is the largest among all other detected faces.
[0024] Another processing task is to generate training sequences. Due to the limited number of available movies and the expensive manual annotation process, the amount of annotated movie data is quite small. Because the time interval between frames is small, the facial differences between one frame and the next are small. In order to obtain more sequences to use in learning, facial sequences for each movie scene can be generated in the following way. For each scene, the frame sequence can be downsampled, for example, at an interval of 18 frames, to obtain a frame sequence. Furthermore, in order to obtain more training data with one scene, all previous sampling indices can be shifted every two frames, and all generated samples from the same video scene share the same emotion label. By doing so, the dynamic changes of the face in each sequence are taken into account, while more training samples are obtained based on the limited video data. This process is as shown in Figure 2 shown.
[0025] Facial expression embedding
[0026] Additional facial expression datasets may be used to generate facial expression embeddings instead of training directly from cropped facial images from movies. As an example and not limitation, a neural network trained with a convolutional neural network (CNN) may be used to classify different facial expression labels.
[0027] RL for emotion label prediction
[0028] In one implementation, a deep Q network (DQN) can be used with an RL framework according to aspects of the present disclosure. The input to the DQN can include two parts: a) facial expression embedding, which can be the output of the second to last layer of a pre-trained CNN of the facial embedding model discussed above; and 2) a predicted emotion information input vector, which includes video frame data at time step t and predicted emotion information from the previous time step t-1, as described above. Two fully connected emotion information input layers are added and concatenated with the facial expression embedding to be further connected with three additional dense layers. At the last layer of the DQN, the network can output discrete two-dimensional actions. In such an implementation, emotion label prediction works as a binary classification decision of an emotion label, for example, a one-hot-coded two-dimensional vector of funny or not funny.
[0029] Many different types of functions G can be used to represent sentiment information. For example, the action embedding output as the output of the penultimate layer of the DQN can be viewed as a high-dimensional feature representation vector with a memory of all previous states. Alternatively, one can directly use the one-hot encoded classification label identified by the DQN output, which shows the action decision label at the previous time step, such as funny or not funny.
[0030] In RL, the setting of the reward function is important because the reward value is used to update the Q value, and the DQN uses the reward (the difference between the score of the current frame t and the score of the previous frame t-1) to learn action decisions. The annotation labels of some implementations may not be available for every frame of the video scene. Instead, for each training sequence sample, only the human annotation labels of the entire training sequence may be available. In each facial sequence, the frame-level emotion label is not available. In such an implementation, the reward generation function H can be designed so that within each frame sequence training sample, the reward value is based on the majority vote of the frame-level results from the last human-annotated frame to the current frame t. The reward value can also be based on the minimum percentage vote of the frame-level results. For example, if 20% or more of the frame-level results are "funny" and the final human label is "funny", the reward may be +1. Based on this design, a reward can be assigned at each frame step, not just at the end of the entire sequence. In an alternative implementation, there may be no reward until the moment when the human annotation is available.
[0031] According to aspects of the present disclosure, the reward value is crucial to the effectiveness of sequence-level prediction using RL. If the reward value is too large or too small, the RL system will not converge and will not be able to learn sentiment information. It is also important to consider the weighting factors of the intermediate and final step rewards.
[0032] For a training sequence, the last action generated by the agent considers all images. In addition, the ground truth labels are available. Therefore, a large reward value R should be assigned end However, at intermediate time steps, there is no guarantee that the pitch label of each frame should be the same as the sequence label.
[0033] This assumption is consistent with common understanding. For example, when a person thinks a movie scene is funny, it is not guaranteed that every frame of the scene contains funny information. Since the sentiment labels are accumulated over the complete sequence of frames, the rewards of the intermediate time steps (denoted as R inter ) should be assigned smaller values to indicate uncertainty and lower priority.
[0034] More details on possible experiments and designed configurations of these features can be found in Haoqi Li, Naveen Kumar, Ruxin Chen, “A DEEP REINFORCEMENT LEARNING FRAMEWORK FOR IDENTIFYING FUNNY SCENES INMOVIES,” 2018 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 3116-3120, which is incorporated herein by reference and constitutes a part of this disclosure as an appendix.
[0035] Method
[0036] Figure 3 The flowchart shown depicts an example of a method for performing sequence-level prediction of a video scene according to various aspects of the present disclosure. The method begins at 302, where video information in a video scene is represented as a sequence of features depicted in each frame. As an example and not limitation, the feature sequence may be a sequence of faces appearing in the video scene. The facial sequence may be a sequence of faces of a single character in the video scene. When multiple faces are displayed in the video scene, the facial sequence may be a sequence of faces located at the center of the scene. Next, as shown in 304, the state of the environment at each time step t corresponding to each frame is represented by the video information at time step t and the predicted emotional information from the previous time step t-1. Then, as shown in 306, the agent controlled by the machine learning algorithm takes an action A(t) for the frame at step t. The output of the action A(t) represents the emotional label prediction of the frame at time step t. As an example and not limitation, the emotional label prediction at time step t may represent whether the video scene is funny or not funny. However, in an alternative implementation, the emotional label prediction may represent whether the video scene is sad or not sad.
[0037] As shown at 308, the predicted action pool (from the initial frame of the scene until the current frame at time step t) is then transformed into, for example, a predicted emotion history for the next time step t+1. As shown at 310, the predicted emotion history is included as part of the environment state for the next time step t+1. As shown at 312, a reward R is generated for the predicted action by comparing the predicted action up to the current time step t with the corresponding annotated movie scene emotion label. In some implementations, the value of the reward R can be based on a majority vote of the frame-level emotion label prediction results up to the current frame at time step t.
[0038] System
[0039] Figure 4 Describes the method used to implement a Figure 3 The system may include a computing device 400 coupled to a user input device 402. The user input device 402 may be a controller, a touch screen, a microphone, a keyboard, a mouse, a joystick, or other device that allows a user to input information, including sound data, to the system. The user input device may be coupled to a tactile feedback device 421. The tactile feedback device 421 may be, for example, a vibration motor, a force feedback system, an ultrasonic feedback system, or a pneumatic feedback system.
[0040] The computing device 400 may include one or more processor units 403, which may be configured according to well-known architectures, such as single-core, dual-core, quad-core, multi-core, processor coprocessor, cell processor, etc. The computing device may also include one or more memory units 404 (e.g., random access memory (RAM), dynamic random access memory (DRAM), read-only memory (ROM), etc.).
[0041] The processor unit 403 may execute one or more programs, portions of which may be stored in the memory 404, and the processor unit 403 may be operably coupled to the memory, for example by accessing the memory via a data bus 405. The memory may include a buffer 408 storing video frames of video scenes. The programs may include a machine learning algorithm 421 configured to extract features 409 from the video frames and assign label predictions 410, as described above. In addition, the memory 404 may contain programs that implement training neural networks. The memory 404 may also contain a database 422 with trained data (e.g., video scenes with known labels). The database 422 may be stored as data 418 in a mass storage device 415, or on a server coupled to a network 420 accessed via a network interface 414.
[0042] The input video may also be stored as data 418 in the mass storage device 415. The processor unit 403 is also configured to execute one or more programs 417 stored in the mass storage device 415 or the memory 404, which cause the processor to perform the method 300 described above.
[0043] The computing device 400 may also include well-known support circuits, such as input / output (I / O) 407, circuits, power supply (P / S) 411, clock (CLK) 412, and cache 413, which may communicate with other components of the system, for example, via bus 405. The computing device may include a network interface 414. The processor unit 403 and the network interface 414 may be configured to implement a local area network (LAN) or PAN via a suitable network protocol such as Bluetooth for a personal area network (PAN). The computing device may optionally include a mass storage device 415, such as a disk drive, CD-ROM drive, tape drive, flash memory, etc., and the mass storage device may store programs and / or data. The computing device may also include a user interface 416 to facilitate interaction between the system and the user. The user interface may include a monitor, a television screen, a speaker, headphones, or other devices that convey information to the user.
[0044] The computing device 400 may include a network interface 414 to facilitate communication via an electronic communications network 420. The network interface 414 may be configured to implement wired or wireless communications over a local area network and a wide area network such as the Internet. The device 400 may send and receive data and / or file requests via one or more message packets over the network 420. Message packets sent over the network 420 may be temporarily stored in a buffer in the memory 404. The classified sound database may be obtained via the network 420 and partially stored in the memory 404 for use.
[0045] Various aspects of the present disclosure address the problem of labeling scenes in videos with facial images using RL methods by modifying traditional RL structures to achieve the desired interaction between the label prediction agent and the environment state. Output emotion labels can be generated in real time using sequence-level predictions of video scenes discussed herein. The described sequence-level predictions attempt to learn strategies through exploration and exploitation, which is achieved by trying different strategies to see if they work better than those tried before. The described sequence-level predictions also have a memory mechanism that allows it to try the strategies that worked best in the past. These properties are not typically achieved through conventional supervised learning methods, which are typically purely exploitative.
[0046] While the above is a complete description of the preferred embodiment of the invention, various alternatives, modifications and equivalents may be used. Therefore, the scope of the invention should not be determined with reference to the above description, but should be determined with reference to the appended claims and their full scope of equivalents. Any feature described herein, whether preferred or non-preferred, may be combined with any other feature described herein, whether preferred or non-preferred. In the appended claims, The indefinite article "a" or "an" Refers to the quantity of one or more of the item following the article unless explicitly stated otherwise. The appended claims should not be interpreted as including means-plus-function limitations unless such a limitation is explicitly recited in a given claim using the phrase "means for..."
Claims
1. A method for performing sequence-level prediction of video scenes, wherein include: Representing video information in the video scene as a sequence of features depicted in each frame; Representing the state of the environment at each time step t corresponding to each frame by the video information at time step t and the predicted emotion information from the previous time step t-1; Using an agent controlled by a machine learning algorithm to take an action A(t) on a frame at time step t, wherein the output of the action A(t) represents a prediction of a sentiment label for the frame at time step t; Convert the predicted action pool into the predicted emotion history for the next time step t+1; including the predicted emotion history as part of the state of the environment at the next time step t+1; and A reward R is generated for the predicted action by comparing the predicted action up to the time step t with the corresponding annotated movie scene emotion label.
2. The method according to claim 1, wherein the emotion label prediction at the time step t indicates whether the video scene is funny or not funny.
3. The method according to claim 1, wherein the value of the reward R is based on a majority vote of the frame-level emotion label prediction results up to the frame at the time step t.
4. The method of claim 1, wherein the value of the reward R is based on a minimum percentage vote of frame-level results.
5. The method of claim 1, wherein the value of the reward R is zero before the frame when the human label is available. The method according to claim 1 , wherein the feature sequence is a sequence of faces appearing in the video scene. The method of claim 6 , wherein the facial sequence is a facial sequence of a single character in the video scene.
8. The method of claim 6, wherein the facial sequence is a facial sequence of a single character in the video scene, and wherein when multiple faces are displayed in the video scene, the facial sequence is a facial sequence located at the center of the scene.
9. A system for performing sequence-level prediction of video scenes, wherein include: processor; a memory coupled to the processor; as well as processor-executable instructions contained in the memory, the instructions being configured to, when executed, cause the processor to: Representing video information in a video scene as a sequence of features depicted in each frame; Representing the state of the environment at each time step t corresponding to each frame by the video information at time step t and the predicted emotion information from the previous time step t-1; Using an agent controlled by a machine learning algorithm to take an action A(t) on a frame at time step t, wherein the output of the action A(t) represents a prediction of a sentiment label for the frame at time step t; Convert the predicted action pool into the predicted emotion history for the next time step t+1; including the predicted emotion history as part of the state of the environment at the next time step t+1; and A reward R is generated for the predicted action by comparing the predicted action up to the time step t with the corresponding annotated movie scene emotion label.
10. The system of claim 9, wherein the emotion label prediction at the time step t indicates whether the video scene is funny or not funny.
11. The system of claim 9, wherein the value of the reward R is based on a majority vote of frame-level sentiment label prediction results up to the frame at the time step t.
12. The system of claim 9, wherein the value of the reward R is based on a minimum percentage vote of frame-level results.
13. The system of claim 9, wherein the value of the reward R is zero prior to a frame when a human label is available.
14. The system of claim 9, wherein the feature sequence is a sequence of faces appearing in the video scene.
15. The system of claim 14, wherein the facial sequence is a facial sequence of a single character in the video scene.
16. The system of claim 14, wherein the facial sequence is a facial sequence of a single character in the video scene, and wherein when multiple faces are displayed in the video scene, the facial sequence is a facial sequence located in the center of the scene.
17. A non-transitory computer-readable medium having executable instructions embodied therein, the instructions being configured to, when executed, cause a computer to perform the following operations: Representing video information in a video scene as a sequence of features depicted in each frame; Representing the state of the environment at each time step t corresponding to each frame by the video information at time step t and the predicted emotion information from the previous time step t-1; Using an agent controlled by a machine learning algorithm to take an action A(t) on a frame at time step t, wherein the output of the action A(t) represents a prediction of a sentiment label for the frame at time step t; Convert the predicted action pool into the predicted emotion history for the next time step t+1; including the predicted emotion history as part of the state of the environment at the next time step t+1; and A reward R is generated for the predicted action by comparing the predicted action up to the time step t with the corresponding annotated movie scene emotion label.
18. The non-transitory computer-readable medium of claim 17, wherein the emotion label prediction at the time step t indicates whether the video scene is funny or not funny.
19. The non-transitory computer-readable medium of claim 17, wherein the value of the reward R is based on a majority vote of frame-level sentiment label prediction results up to the frame at the time step t.
20. The non-transitory computer-readable medium of claim 17, wherein the value of the reward R is based on a minimum percentage vote of frame-level results.
21. The non-transitory computer readable medium of claim 17, wherein the value of the reward R is zero prior to a frame when a human label is available.
22. The non-transitory computer-readable medium of claim 17, wherein the sequence of features is a sequence of faces appearing in the video scene.
23. The non-transitory computer-readable medium of claim 22, wherein the facial sequence is a facial sequence of a single character in the video scene.
24. The non-transitory computer-readable medium of claim 22, wherein the facial sequence is a facial sequence of a single character in the video scene, and wherein when multiple faces are displayed in the video scene, the facial sequence is a facial sequence located in the center of the scene.
