Information processing device and information processing method

Abstract

The present invention enables user's emotion for each scene of moving image content to be effectively used.　The present invention generates, on the basis of user's emotion and video quality for each scene of moving image content A, correlation data obtained by associating the user's emotion and the video quality with each other. The present invention predicts, on the basis of video quality for each scene of moving image content B and the correlation data obtained by associating the user's emotion and the video quality related to moving image content A, the user's emotion for each scene of moving image content B. For example, the predicted user's emotion for each scene of moving image content B is displayed and used.

Description

Technical Field

[0001] This technology relates to an information processing apparatus and an information processing method, and more particularly to an information processing apparatus that processes information related to video content.

Background Art

[0002] Conventionally, various technologies have been proposed for generating emotion data indicating user emotions for each scene of video content based on a user's face image, user's biometric information, etc. (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] An object of this technology is to effectively utilize user emotions for each scene of video content.

Means for Solving the Problems

[0005] The concept of this technology is an information processing apparatus including a data generation unit that generates correlation data linking user emotions and video quality based on user emotions for each scene of video content and video quality.

[0006] ​In this technology, the data generation unit generates correlation data that links user emotions and video quality based on user emotions and video quality for each scene of the video content. For example, the correlation data may consist of combination data of user emotions and video quality for each scene. In this case, since there will be a large amount of correlation data of combination data of user emotions and video quality, it will be possible to accurately calculate, for example, user emotions corresponding to video quality.

[0007] Furthermore, for example, the correlation data may be regression equation data calculated based on combination data of user sentiment and video quality for each scene. In this case, since the correlation data is regression equation data, it is possible to save storage capacity in the database storing this correlation data, and it becomes possible to easily calculate user sentiment corresponding to, for example, video quality. In this case, for example, correlation coefficient data may be added to the regression equation data. Based on this correlation coefficient data, it becomes possible to decide whether or not to use the regression equation. Also, for example, the data generation unit may generate correlation data for each user attribute using user sentiment for each user attribute. This makes it possible to selectively use correlation data for desired attributes.

[0008] Thus, this technology generates correlation data linking user emotions and video quality based on user emotions and video quality for each scene of video content, making it possible to obtain good correlation data linking user emotions and video quality.

[0009] Furthermore, other concepts of this technology include: The system includes a user emotion prediction unit that predicts user emotions for each scene of the video content based on correlation data linking the video quality of each scene of the video content with user emotions. It is located in the information processing unit.

[0010] In this technology, the user emotion prediction unit predicts the user's emotion for each scene of the video content based on the video quality for each scene of the video content and correlation data linking user emotion with video quality. For example, the user emotion prediction unit may predict the user's emotion for each scene of the video content based on correlation data of predetermined attributes selected from correlation data for each user attribute. This makes it possible for the user emotion prediction unit to obtain emotion data suitable for the user's desired attributes and use it for playing or editing the video content.

[0011] In this technology, the user's emotions for each scene of a video content are predicted based on correlation data linking the video quality of each scene with user emotions, making it possible to accurately predict user emotions for each scene of a video content.

[0012] Furthermore, this technology may include, for example, a display control unit that controls the display of user emotions for each scene of the predicted video content. This allows the user to easily recognize the predicted user emotions for each scene of the video content, and to easily and effectively perform selective playback operations on the video content, as well as editing operations such as selective extraction and image quality correction of the video content.

[0013] Furthermore, this technology may also include an extraction unit that extracts emotionally representative scenes based on the user's emotions towards each scene of the predicted video content. This makes it possible to effectively utilize the user's emotions towards each scene of the predicted video content during video content playback and editing.

[0014] For example, the extraction unit may extract emotionally representative scenes based on the type of user emotion. Alternatively, for example, the extraction unit may extract emotionally representative scenes based on the degree of user emotion. In this case, for example, the extraction unit may extract scenes where the degree of user emotion exceeds a threshold as emotionally representative scenes. Alternatively, for example, the extraction unit may extract emotionally representative scenes based on a statistical value of the overall degree of user emotion in the video content. Here, the statistical value may include, for example, the maximum value, sorting result, mean, or standard deviation.

[0015] Furthermore, this technology may also include a playback control unit that controls the playback of video content based on extracted emotionally representative scenes. This allows the user to view only the extracted emotionally representative scenes, or only the remaining parts excluding the extracted emotionally representative scenes. The information processing apparatus according to claim 6.

[0016] Furthermore, this technology may also include an editing control unit that controls the editing of video content based on extracted emotional representative scenes. This allows the user to obtain new video content that includes only the extracted emotional representative scenes, or only the remaining parts excluding the extracted emotional representative scenes, or to obtain new video content in which the video quality of only the extracted emotional representative scenes, or the remaining parts excluding the extracted emotional representative scenes, has been corrected. [Brief explanation of the drawing]

[0017] [Figure 1] This block diagram shows an example configuration of an information processing device that generates emotion metadata. [Figure 2] This is a block diagram showing an example configuration of an information processing device that generates correlation data linking user emotions and video quality. [Figure 3]It is a diagram showing an example of video quality information and user emotion information for each frame of video content A. [Figure 4] It is a scatter diagram showing correlation data consisting of combined data of user emotion and video quality for each frame. [Figure 5] It is a diagram showing another example of video quality information and user emotion information for each frame of video content A. [Figure 6] It is a scatter diagram showing other correlation data consisting of combined data of user emotion and video quality for each frame. [Figure 7] It is a diagram for explaining the case where the correlation data is data of a regression equation calculated based on combined data of user emotion and video quality for each frame. [Figure 8] It is a block diagram showing a configuration example of an information processing apparatus that uses correlation data linking user emotion and video quality. [Figure 9] It is a diagram showing an example of UI display displayed on the display unit of the content playback / editing unit. [Figure 10] It is a diagram showing another example of UI display displayed on the display unit of the content playback / editing unit. [Figure 11] It is a block diagram showing a configuration example of another information processing apparatus that uses correlation data linking user emotion and video quality. [Figure 12] It is a diagram for explaining the case of extracting a scene where the degree of user emotion exceeds a threshold as an emotion representative scene. [Figure 13] It is a diagram for explaining the case of extracting an emotion representative scene based on a statistical value of the degree of overall user emotion of video content.

Mode for Carrying Out the Invention

[0018] Hereinafter, a mode for carrying out the invention (hereinafter referred to as "embodiment") will be described. The description will be made in the following order. 1. Embodiment 2. Variation

[0019] <1. Embodiment> This technology includes the steps of generating emotion data indicating user emotions for each scene of a first video content (video content A), generating correlation data linking user emotions and video quality based on user emotions and video quality for each scene of the first video content (video content A), and predicting and utilizing user emotions for each scene of a second video content (video content B).

[0020] [Example configuration of an information processing device that generates emotional metadata] Figure 1 shows an example configuration of an information processing device 100 that generates emotion metadata. This information processing device 100 includes a content database (content DB) 101, a content playback unit 102, a face image capture camera 103, a biometric information sensor 104, a user emotion analysis unit 105, a metadata generation unit 106, and a metadata database (emotion data DB) 107.

[0021] The content database 101 stores multiple video content files. When a playback video file name (video content A) is input to the content database 101, it supplies the content playback unit 102 with a video content file containing video content A that corresponds to the playback video file name. Here, the playback video file name is specified, for example, by the user of this information processing device 100.

[0022] During playback, the content playback unit 102 plays video content A contained in a video content file supplied from the content database 101 and displays the video on a display unit (not shown). Furthermore, during playback, the content playback unit 102 supplies frame numbers (timecodes) to the metadata generation unit 106 in synchronization with the playback frames. These frame numbers are information that can identify a scene in video content A.

[0023] The face image capture camera 103 is a camera that captures the face image of the user watching the video displayed on the display unit by the content playback unit 102. The face images of each frame captured by this face image capture camera 103 are sequentially supplied to the user emotion analysis unit 105.

[0024] The biometric information sensor 104 is attached to the user who is watching the video displayed on the display unit of the content playback unit 102, and is a sensor for acquiring biometric information such as heart rate, respiratory rate, and sweating amount. The biometric information of each frame acquired by this biometric information sensor 104 is sequentially supplied to the user emotion analysis unit 105.

[0025] The user emotion analysis unit 105 analyzes the degree of a predetermined type of user emotion for each frame based on the face images of each frame sequentially supplied from the face image capture camera 103 and the biometric information of each frame sequentially supplied from the biometric information sensor 104, and supplies the user emotion information to the metadata generation unit 106.

[0026] Furthermore, the types of user emotions are not limited to secondary information obtained by analyzing facial images and biometric data, such as information on "joy," "anger," "sadness," and "happiness," but may also be primary information such as heart rate, respiratory rate, and sweating volume.

[0027] The metadata generation unit 106 associates the user sentiment information for each frame obtained by the user sentiment analysis unit 105 with the frame number (time code) to generate sentiment metadata containing user sentiment information for each frame of the video content A, and supplies this sentiment metadata to the metadata database 107.

[0028] The metadata database 107 stores sentiment metadata corresponding to multiple video content files. The metadata database 107 stores the sentiment metadata supplied from the metadata generation unit 106 in a database along with the video file name, in other words, it stores it linked to the video file name, so that it can identify which video content file the sentiment metadata corresponds to.

[0029] If the emotion metadata corresponding to the playback video file name (video content A) has not yet been stored, the emotion metadata supplied by the metadata generation unit 106 is stored as is. If the metadata database 107 already stores the emotion metadata corresponding to the playback video file name (video content A), it is updated with the emotion metadata supplied by the metadata generation unit 106.

[0030] Alternatively, if the metadata database 107 already stores emotion metadata corresponding to the playback video file name (video content A), it updates the already stored emotion metadata with emotion metadata obtained by combining it with emotion metadata supplied from the metadata generation unit 106.

[0031] A weighted average is a possible method for synthesis, but it is not limited to this and other methods may also be used. In the case of a weighted average, if the sentiment metadata already attached pertains to m users, the already attached sentiment metadata and the sentiment metadata supplied from the metadata generation unit 106 are weighted m:1 and averaged.

[0032] When updating with sentiment metadata obtained through this synthesis, the more users who view video content A, the more the sentiment metadata is updated, resulting in more accurate sentiment metadata. In this case, sentiment metadata generated by one user's viewing will contain the sentiment information of that one user, while sentiment metadata generated by multiple users' viewing will contain sentiment information that is statistically representative of the sentiment responses of the other users.

[0033] Furthermore, when generating sentiment metadata, instead of multiple users sequentially viewing video content and updating the sentiment metadata, it is also conceivable to obtain highly accurate sentiment metadata all at once by inputting facial images and biometric information related to multiple users into the user sentiment analysis unit 105 and performing analysis.

[0034] In the illustrated example, the association between sentiment metadata stored in the metadata database 107 and video content files stored in the content database 101 is shown to be performed using the video file name. However, other methods may also be used, such as recording link information, such as a URL for accessing the sentiment metadata stored in the metadata database 107, as metadata within the corresponding video content file in the content database 101.

[0035] As shown in Figure 1, the information processing device 100 generates emotion metadata containing user emotion information for each frame of video content, and stores this emotion metadata in the metadata database 107, linked to the video content file. This makes it possible to easily use the emotion metadata linked to the video content file, for example.

[0036] [Example configuration of an information processing device that generates correlated data] Figure 2 shows an example configuration of an information processing device 200 that generates correlation data linking user emotions and video quality. This information processing device 200 includes a content database (content DB) 201, a content playback unit 202, a video quality analysis unit 203, a metadata database (metadata DB) 204, a correlation data generation unit 205, and a metadata database (metadata DB) 206.

[0037] The content database 201 corresponds to the content database 101 shown in Figure 1 and stores multiple video content files. When a playback video file name (video content A) is input to the content database 201, it supplies the video content file corresponding to that playback video file name to the content playback unit 202. Here, the playback video file name is specified, for example, by the user of this information processing device 200.

[0038] The content playback unit 202 plays video content A contained in the video content file supplied from the content database 201 and supplies the video signal related to video content A to the video quality analysis unit 203.

[0039] The video quality analysis unit 203 analyzes the degree of camera shake (remaining correction), zoom speed, focus deviation, etc., for each frame based on the video signal of each frame supplied from the content playback unit 202, and obtains video quality data containing video quality information for each frame of video content A, which is supplied to the correlation data generation unit 205. Here, the video quality information may consist of multiple primary pieces of information in parallel, such as camera shake (remaining correction), zoom speed, and focus deviation, or it may be a single piece of video quality information obtained as secondary information by integrating these multiple primary pieces of information.

[0040] For example, the video quality analysis unit 203, although a detailed explanation is omitted, uses well-known machine learning and AI (Artificial Intelligence) technologies to determine the video quality frame by frame for content to be evaluated in advance. It should be noted that even without using machine learning or AI technologies, it is possible to calculate some evaluation value dependent on quality using a simple filter configuration.

[0041] Metadata database 204 corresponds to metadata database 107 shown in Figure 1 and stores sentiment metadata associated with multiple video content files stored in content database 201. In this example, the association is shown using the video file name.

[0042] When the metadata database 204 receives the same playback video file name (video content A) as the one entered into the content database 201, it supplies the correlation data generation unit 205 with sentiment metadata that contains user sentiment information for each frame of video content A, which is linked to the video content file supplied from the content database 201 to the content playback unit 202.

[0043] The correlation data generation unit 205 generates correlation data linking user sentiment and video quality based on video quality data supplied from the video quality analysis unit 203 and sentiment metadata supplied from the metadata database 204, that is, based on user sentiment and video quality for each frame of video content A, and supplies this correlation data to the metadata database 206.

[0044] This correlation data consists, for example, of combination data of user sentiment and video quality for each frame.

[0045] Figure 3 shows an example of video quality information and user sentiment information for each frame of video content A. Figure 3(a) shows the video quality information. In this example, the video quality information consists of three pieces of information (primary information): the amount of camera shake (remaining correction), the zoom speed, and the degree of focus shift. Figure 3(b) shows the user sentiment information for each frame of video content A. In this example, the sentiment information consists of three pieces of information (primary information): heart rate, skin temperature, and sweating amount.

[0046] Figure 4 shows the correlation data in that case as a scatter plot. In this case, the correlation data includes combinations of each of the following for each frame: camera shake amount (remaining correction), zoom speed, and focus shift, as well as heart rate, skin temperature, and sweating amount. Note that in Figure 4, the display of points representing combination data is omitted in the scatter plots except for the combination data of camera shake amount (remaining correction) and heart rate for each frame.

[0047] Figure 5 shows another example of video quality information and user sentiment information for each frame of video content A. Figure 5(a) shows the video quality information. In this example, the video quality information consists of one piece of video quality information (secondary information) obtained by integrating multiple pieces of information, such as the amount of camera shake (remaining correction), zoom speed, and focus deviation mentioned above. 。 Figure 5(b) shows user emotion information for each frame of video content A. In this example, the emotion information consists of four pieces of information (secondary information), such as "joy," "anger," "sadness," and "happiness."

[0048] Figure 6 shows the correlation data in that case as a scatter plot. In this case, the correlation data consists of combinations of the image quality level for each frame and the four levels: "joy," "anger," "sadness," and "happiness." Note that in Figure 6, the points representing the combination data are omitted in the scatter plots other than the combination data of the image quality level and the "joy" level for each frame.

[0049] In the above examples, we showed cases where both video quality information and user sentiment information are either primary or secondary information. However, it is not limited to primary information with primary information, or secondary information with secondary information, but can also be combinations or cross-sections of these.

[0050] The above example showed correlation data consisting of combinations of user sentiment and video quality for each frame. In this case, since the correlation data will contain a large number of combinations of user sentiment and video quality, it becomes possible to accurately calculate, for example, user sentiment corresponding to video quality.

[0051] However, correlation data can also be data from a regression equation calculated based on combination data of user sentiment and video quality for each frame. For example, Figure 7(a) shows a scatter plot of combination data of user sentiment (y) and video quality (x) for each frame. Figure 7(b) shows an example of a regression equation (linear function) and correlation coefficient obtained by degenerating this combination data using a general statistical method. In this case, the slope a, intercept b, and correlation coefficient r are stored as correlation data.

[0052] Figure 7(c) shows the use of the regression equation. By using this regression equation, it is possible to determine user sentiment (y) from image quality (x). In this case, if the correlation coefficient r is small, it is unreliable and should not be used, or if the correlation coefficient r is large, it should be used actively.

[0053] By using correlation data as the data for the regression equation in this way, it becomes possible to save storage space in the database that stores this correlation data, and it also becomes possible to easily calculate, for example, user sentiment corresponding to video quality. Furthermore, by adding correlation coefficient data to the regression equation data, it becomes possible to easily and appropriately decide whether or not to use the regression equation.

[0054] Returning to Figure 2, the metadata database 206 stores correlation metadata corresponding to multiple video content files. The metadata database 206 databases the correlation data supplied from the correlation data generation unit 205, along with the video file name, so that it can identify which video content file the sentiment metadata corresponds to. Link information such as URLs for accessing the correlation data stored in the metadata database 206 may also be recorded as metadata within the corresponding video content file in the content database 201.

[0055] As shown in Figure 2, the information processing device 200 generates correlation data linking user emotions and video quality based on user emotions and video quality for each scene of video content A, making it possible to obtain good correlation data linking user emotions and video quality.

[0056] [Example configuration of an information processing device that utilizes correlated data] Figure 8 shows an example configuration of an information processing device 300 that utilizes correlation data linking user sentiment and video quality. This information processing device 300 includes a content database (content DB) 301, a content playback unit 302, a video quality analysis unit 303, a metadata database (metadata DB) 304, a user sentiment prediction unit 305, and a content playback / editing unit 306.

[0057] The content database 301 stores multiple video content files. When a playback video file name (video content B) is input to the content database 301, it supplies the video content file corresponding to that playback video file name to the content playback unit 302 and the content playback / editing unit 306. Here, the playback video file name is specified, for example, by the user of this information processing device 300.

[0058] The content playback unit 302 plays video content B contained in the video content file supplied from the content database 301 and supplies the video signal related to video content B to the video quality analysis unit 303.

[0059] The video quality analysis unit 303 is configured similarly to the video quality analysis unit 203 shown in Figure 2. Based on the video signal of each frame supplied from the content playback unit 302, it analyzes the degree of camera shake (remaining correction), zoom speed, focus deviation, etc. for each frame to obtain video quality data containing video quality information for each frame of video content A, and supplies it to the user emotion prediction unit 305.

[0060] The metadata database 304 corresponds to the metadata database 206 shown in Figure 2 and stores correlation data linking user sentiment and video quality for multiple video content files. When the playback video file name (video content A) is input to the metadata database 304, it supplies correlation data corresponding to video content A to the user sentiment prediction unit 305.

[0061] The user sentiment prediction unit 305 predicts the user sentiment for each frame of video content B based on the video quality for each frame of video content B and correlation data that links user sentiment and video quality corresponding to video content A. It then obtains sentiment data containing user sentiment information for each frame of video content B and supplies it to the content playback / editing unit 306.

[0062] The content playback / editing unit 306, in response to user operations, performs control (not shown) to selectively play a portion of video content B, selectively extract a portion of video content B contained in a video content file, or selectively correct the video quality of a portion of video content B to generate new video content C.

[0063] As described above, the emotion data obtained by the user emotion prediction unit 305 contains user emotion information for each frame of video content B, indicating what emotions the viewer feels for each frame of video content B. In the content playback / editing unit 306, a control unit (not shown) generates, for example, a UI (User Interface) that displays user emotion information for each frame of video content B based on the emotion data. of Display control is performed to assist the user in selectively playing video content B, selectively extracting video content B, and performing editing operations to generate new video content C by correcting the video quality.

[0064] Figure 9 shows an example of the UI display shown on the display unit 361 of the content playback / editing unit 306. In this example, there is a display area 362 at the bottom that displays user emotion information (heart rate, skin temperature, sweat amount) for each frame of video content B, corresponding to a time axis slider bar that shows the progress of video content playback, and there is a display area 363 at the top that displays the playback video.

[0065] Figure 10 shows another example of the UI display shown on the display unit 361 of the content playback / editing unit 306. In this example, there is a display area 364 at the bottom that displays user emotion information (heart rate, skin temperature, sweating amount) for each frame of video content B, and further displays video quality information (amount of camera shake (remaining correction), zoom speed, focus deviation) for each frame of video content B, corresponding to a time axis slider bar that shows the progress of video content playback, and there is a display area 363 at the top that displays the playback video. In this case, as shown by the dashed line in Figure 8, video quality data obtained by the video quality analysis unit 303 is supplied to the content playback / editing unit 306, and video quality information for each frame of video content B is displayed based on this video quality data.

[0066] As shown in Figure 8, the information processing device 300 uses a user emotion prediction unit 305 to predict user emotions for each frame of video content B based on correlation data that links the video quality for each frame of video content B with user emotions related to video content A and the video quality. This makes it possible to predict user emotions for each frame of video content B accurately.

[0067] Furthermore, in the information processing device 300 shown in Figure 8, the content playback / editing unit 306 displays the user's emotions for each scene of the video content B based on emotion data containing user emotion information for each frame of the video content B obtained by the user emotion prediction unit 305. This allows the user to easily recognize the predicted user emotions for each frame of the video content B, and to easily and effectively perform selective playback operations on the video content B, as well as editing operations such as selective extraction and image quality correction of the video content B.

[0068] Furthermore, in the information processing device 300 shown in Figure 8, by inputting the newly generated video content C in the content playback / editing unit 306 as equivalent to video content B, the user emotion prediction unit 305 can predict user emotions for each frame of video content C. This can be used to check the completeness of video content C, leading to the completion of higher-quality video content and assisting creators in their creative activities.

[0069] [Other configuration examples of information processing devices that utilize correlated data] Figure 11 shows an example configuration of an information processing device 300A that utilizes correlation data linking user emotions and video quality. In Figure 11, parts corresponding to those in Figure 8 are denoted by the same reference numerals, and their detailed explanations are omitted as appropriate.

[0070] This information processing device 300A includes a content database (content DB) 301, a content playback unit 302, a video quality analysis unit 303, a metadata database (metadata DB) 304, a user emotion prediction unit 305, an emotion representative scene extraction unit 311, and a content playback / editing unit 312.

[0071] The content database 301, upon input of a playback video file name (video content B), supplies the video content file corresponding to that playback video file name to the content playback unit 302 and the content playback / editing unit 312. The metadata database 304, upon input of a playback video file name (video content A), supplies correlation data corresponding to video content A to the user sentiment prediction unit 305.

[0072] The content playback unit 302 plays video content B contained in the video content file supplied from the content database 301 and supplies the video signal related to video content B to the video quality analysis unit 303. Based on the video signal of each frame supplied from the content playback unit 302, the video quality analysis unit 303 analyzes the degree of camera shake (remaining correction), zoom speed, focus deviation, etc. for each frame to obtain video quality data containing video quality information for each frame of video content A and supplies it to the user emotion prediction unit 305.

[0073] The user emotion prediction unit 305 predicts the user emotion for each frame of video content B based on the video quality for each frame of video content B and correlation data that links user emotion and video quality corresponding to video content A. It then obtains emotion data containing user emotion information for each frame of video content B and supplies it to the emotion representative scene extraction unit 311.

[0074] The emotion representative scene extraction unit 311 extracts emotion representative scenes from the emotion metadata supplied by the user emotion prediction unit 305.

[0075] For example, the emotion representative scene extraction unit 311 extracts emotion representative scenes based on the type of user emotion. In this case, for example, if the emotion metadata contains information on "joy," "anger," "sadness," and "happiness" as user emotion information for each frame of the video content, it selects one of these emotions and extracts scenes where the degree (level) of that emotion is above a threshold as emotion representative scenes. Here, the selection of emotions and the setting of thresholds can be arbitrarily performed, for example, by user operation.

[0076] Furthermore, for example, the emotion representative scene extraction unit 311 extracts emotion representative scenes based on the degree of user emotion. In this case, possible methods include (1) extracting scenes in which the degree of user emotion exceeds a threshold as emotion representative scenes, or (2) extracting emotion representative scenes based on a statistical value of the overall degree of user emotion in the video content.

[0077] First, we will explain the case where (1) scenes in which the degree of user emotion exceeds a threshold are extracted as representative emotional scenes. In this case, for example, if the emotion metadata contains information on "joy," "anger," "sadness," and "happiness" as user emotion information for each frame of the video content, then for each emotion, scenes in which the degree (level) exceeds the threshold will be extracted as representative emotional scenes. Here, the threshold can be set arbitrarily, for example, by user operation.

[0078] Figure 12(a) shows an example of the frame-by-frame change in a given degree (level) of user emotion. Here, the horizontal axis represents the frame number fr, and the vertical axis represents the degree of user emotion Em(fr). In this example, since the degree Em(fr_a) exceeds the threshold th at frame number fr_a, frame number fr_a is stored as emotion representative scene information L(1), and since the degree Em(fr_b) exceeds the threshold th at frame number fr_b, frame number fr_b is stored as emotion representative scene information L(2).

[0079] The flowchart in Figure 12(b) shows an example of the processing procedure of the emotion representative scene extraction unit 311 when extracting scenes in which the degree of user emotion exceeds a threshold as emotion representative scenes.

[0080] First, the emotion representative scene extraction unit 311 starts processing in step ST1. Next, in step ST2, the emotion representative scene extraction unit 311 initializes frame numbers fr=1 and n=1.

[0081] Next, in step ST3, the emotion representative scene extraction unit 311 determines whether the degree Em(fr) is greater than the threshold th. If Em(fr) > th, in step ST4, the emotion representative scene extraction unit 311 stores the emotion representative scene information, that is, stores the frame number fr as the emotion representative scene L(n). Also in step ST4, the emotion representative scene extraction unit 311 increments n to n+1.

[0082] Next, in step ST5, the emotion representative scene extraction unit 311 updates the frame number fr to fr = fr + 1. Similarly, if Em(fr) > th is not true in step ST3, the frame number fr is updated in step ST5.

[0083] Next, in step ST6, the emotion representative scene extraction unit 311 determines whether the frame number fr is greater than the last frame number fr_end, that is, it determines the end of the process. If fr > fr_end is not true, the emotion representative scene extraction unit 311 returns to the process in step ST3 and repeats the same process as described above. On the other hand, if fr > fr_end, the emotion representative scene extraction unit 311 terminates the process in step ST7.

[0084] Next, we will explain (2) the case where representative emotional scenes are extracted based on statistical values ​​of the overall degree of user emotion in the video content. In this case, the statistical values ​​may be the maximum value, sorting result, mean, or standard deviation.

[0085] When the statistical value is at its maximum, for example, if the emotion metadata contains information on user emotions such as "joy," "anger," "sadness," and "happiness" for each frame of the video content, the scene where the degree (level) of each emotion is at its maximum will be extracted as the representative scene for that emotion.

[0086] Furthermore, when statistical values ​​are sorting results, for example, if the emotion metadata contains information on user emotions such as "joy," "anger," "sadness," and "happiness" for each frame of video content, then for each emotion, not only the scene with the highest degree (level) but also the scenes ranked second, third, and so on will be extracted as representative scenes of that emotion.

[0087] Furthermore, when statistical values ​​are the mean or standard deviation, for example, if the emotional metadata contains information on user emotions such as "joy," "anger," "sadness," and "happiness" for each frame of video content, then scenes where the degree (level) of each emotion deviates significantly from the average (for example, three times the standard deviation) will be extracted as representative scenes of that emotion.

[0088] Figure 13(a) shows an example of the frame-by-frame change in a given degree (level) of user emotion. Here, the horizontal axis represents the frame number fr, and the vertical axis represents the degree of user emotion Em(fr). In this example, the degree Em(fr_a) for frame number fr_a becomes the maximum value em_max, so frame number fr_a is stored as the emotion representative scene information L.

[0089] The flowchart in Figure 13(b) shows an example of the processing procedure of the emotion representative scene extraction unit 311 when extracting a scene that represents the maximum level of user emotion in the overall video content as an emotion representative scene.

[0090] First, the emotion representative scene extraction unit 311 starts processing in step ST11. Next, in step ST12, the emotion representative scene extraction unit 311 initializes the frame number fr=1 and the maximum value em_max=0.

[0091] Next, in step ST13, the emotion representative scene extraction unit 311 determines whether the degree Em(fr) is greater than the maximum value em_max. If Em(fr) > em_max, in step ST14, the emotion representative scene extraction unit 311 stores the emotion representative scene information, that is, stores the frame number fr as the emotion representative scene L. Also in step ST14, the emotion representative scene extraction unit 311 updates em_max to Em(fr).

[0092] Next, in step ST15, the emotion representative scene extraction unit 311 updates the frame number fr to fr = fr + 1. Similarly, if Em(fr) > em_max is not true in step ST13, the frame number fr is updated in step ST15.

[0093] Next, in step ST16, the emotion representative scene extraction unit 311 determines whether the frame number fr is greater than the last frame number fr_end, that is, it determines the end of the process. If fr > fr_end is not true, the emotion representative scene extraction unit 311 returns to the process in step ST13 and repeats the same process as described above. On the other hand, if fr > fr_end, the emotion representative scene extraction unit 311 terminates the process in step ST17.

[0094] Returning to Figure 11, the emotion representative scene extraction unit 311 supplies emotion representative scene information to the content playback / editing unit 312. In the content playback / editing unit 312, a control unit (not shown) performs control to selectively play a portion of the video content B contained in the video content file supplied from the content database 301, based on the emotion representative scene information supplied from the emotion representative scene extraction unit 311. In this case, for example, depending on the user settings, only the emotion representative scenes can be played, or other parts excluding the emotion representative scenes can be played.

[0095] Furthermore, in the content playback / editing unit 312, a control unit (not shown) performs control to selectively extract a portion of the video content B contained in the video content file supplied from the content database 301, based on the emotional representative scene information supplied from the emotional representative scene extraction unit 311, and generate new video content C. In this case, for example, depending on the user settings, only the emotional representative scenes can be extracted, or other parts excluding the emotional representative scenes can be extracted.

[0096] Furthermore, in the content playback / editing unit 312, a control unit (not shown) performs control to selectively correct the video quality of a portion of the video content B contained in the video content file supplied from the content database 301, based on the emotion representative scene information supplied from the emotion representative scene extraction unit 311, in order to generate new video content C.

[0097] Furthermore, the content playback / editing unit 312 may use not only the emotional representative scene information supplied by the emotional representative scene extraction unit 311, but also other evaluation values ​​that have been used conventionally. Alternatively, as shown by the dashed line in Figure 11, the content playback / editing unit 312 may use not only the emotional representative scene information supplied by the emotional representative scene extraction unit 311, but also video quality data from the video quality analysis unit 303 as an evaluation value.

[0098] As shown in Figure 11, the information processing device 300A extracts emotional representative scenes from the emotional representative scene extraction unit 311 based on the user's emotions for each scene of the predicted video content B. This makes it possible to effectively utilize the user's emotions for each scene of the predicted video content B when playing or editing the video content.

[0099] For example, when a creator creates new video content C from video content B, it becomes possible to automatically perform editing based on scenes that viewers are likely to like or dislike. In other words, creators can perform editing based on these indicators, which ultimately helps in the production of high-quality video content C.

[0100] <2. Variant> Although not mentioned above, it is also conceivable that the information processing device 100 (see Figure 1) generates sentiment metadata by attribute, such as generation, gender, and country, and the information processing device 200 (see Figure 2) generates attribute-specific correlation data using this attribute-specific sentiment data. Furthermore, the information processing devices 300 and 300A (see Figures 8 and 11) can be configured to supply correlation data of predetermined attributes selected by the user, for example using a UI, from the metadata database 304 to the user sentiment prediction unit 305. In this case, the user sentiment prediction unit 305 of the information processing devices 300 and 300A predicts the user's sentiment for each scene of the video content based on the correlation data of the predetermined attributes. This makes it possible for the user sentiment prediction unit 305 to obtain sentiment data suitable for the desired attributes and use it for playback and editing of the video content B.

[0101] Furthermore, in the above-described embodiment, video content A was explained as being a single piece of content. However, video content A may consist of multiple pieces of content. In that case, in the information processing device 200 of Figure 2, a single correlation data will be generated for multiple pieces of video content, and the statistical quality of the correlation data will be improved.

[0102] Furthermore, in the above-described embodiment, an example was shown in which each scene is composed of one frame. However, each scene may be composed of multiple frames.

[0103] Furthermore, while preferred embodiments of this disclosure have been described in detail with reference to the accompanying drawings, the technical scope of this disclosure is not limited to such examples. It is clear to any person with ordinary skill in the art of this disclosure that various modifications or alterations may be conceived within the scope of the technical idea set forth in the claims, and these too will naturally fall within the technical scope of this disclosure.

[0104] Furthermore, the effects described herein are merely descriptive or illustrative and not limiting. In other words, the technology relating to this disclosure may produce other effects that will be apparent to those skilled in the art from the description herein, in addition to or in lieu of the effects described herein.

[0105] Furthermore, this technology can also be configured as follows: (1) The system includes a data generation unit that generates correlation data linking user emotions and video quality based on user emotions and video quality for each scene of the video content. Information processing device. (2) The correlation data consists of combination data of user sentiment and video quality for each scene. The information processing device described in (1) above. (3) The correlation data is the regression equation data calculated based on the combination data of user sentiment and video quality for each scene. The information processing device described in (1) above. (4) The data for the regression equation is accompanied by data for the correlation coefficient. The information processing device described in (3) above. (5) The data generation unit generates the correlation data for each user attribute using the user sentiment for each user attribute. An information processing device as described in any of (1) to (4) above. (6) The procedure for generating correlation data that links user emotions and video quality based on user emotions and video quality for each scene of the video content. Information processing methods. (7) The system includes a user emotion prediction unit that predicts user emotions for each scene of the video content based on correlation data linking the video quality for each scene of the video content with user emotions. Information processing device. (8) The system further comprises a display control unit that controls the display of user emotions for each scene of the predicted video content. The information processing device described in (7) above. (9) The system further comprises an extraction unit that extracts emotionally representative scenes based on the user's emotions towards each scene of the predicted video content. The information processing device described in (7) above. (10) The extraction unit extracts the representative emotion scene based on the type of user emotion. The information processing device described in (9) above. (11) The extraction unit extracts the emotional representative scene based on the degree of the user's emotion. The information processing device described in (9) above. (12) The extraction unit extracts scenes in which the degree of user emotion exceeds a threshold as the representative emotion scene. The information processing device described in (11) above. (13) The extraction unit extracts the emotional representative scenes based on the statistical value of the overall degree of user emotion in the video content. The information processing device described in (11) above. (14) The statistical values ​​include the maximum value, sorting result, mean, or standard deviation. The information processing device described in (13) above. (15) The user emotion prediction unit predicts user emotion for each scene of the video content based on correlation data of predetermined attributes selected from user attribute-specific correlation data. An information processing device as described in any of (7) to (14) above. (16) Further comprising a playback control unit that controls the playback of the video content based on the extracted emotional representative scenes An information processing device as described in any of (7) to (15) above. (17) Further comprising an editing control unit that controls the editing of the video content based on the extracted emotional representative scenes An information processing device according to any of (7) to (16) above. (18) A procedure for predicting user emotions for each scene of a video content based on correlation data linking the video quality for each scene of the video content with user emotions. Information processing methods. [Explanation of symbols]

[0106] 100... Information Processing Device 101. Content Database (Content DB) 102...Content Playback Department 103...Face image capture camera 104... Biometric Information Sensor 105...User Sentiment Analysis Department 106...Metadata generation unit 107. Metadata Database (Metadata DB) 200... Information Processing Device 201...Content Database (Content DB) 202...Content Playback Department 203...Video Quality Analysis Department 204. Metadata Database (Metadata DB) 205...Correlation Data Generation Unit 206. Metadata Database (Metadata DB) 300, 300A... Information Processing Equipment 301...Content Database (Content DB) 302...Content Playback Department 303...Video Quality Analysis Department 304. Metadata Database (Metadata DB) 305...User Sentiment Prediction Department 306...Content Playback / Editorial Department 311... Emotional Representative Scene Extraction Section 312...Content Playback / Editorial Department

Claims

1. The system includes a data generation unit that generates correlation data linking user emotions and video quality based on the user emotions of the user viewing the video content and the video quality for each scene of the video content. Information processing device.

2. The aforementioned correlation data consists of combination data of user sentiment and video quality for each of the aforementioned scenes. The information processing apparatus according to claim 1.

3. The aforementioned correlation data is regression data calculated based on combination data of user sentiment and video quality for each of the aforementioned scenes. The information processing apparatus according to claim 1.

4. The regression equation data is accompanied by correlation coefficient data. The information processing apparatus according to claim 3.

5. The data generation unit generates the correlation data for each user attribute using the user sentiment data for each user attribute. The information processing apparatus according to claim 1.

6. The procedure includes generating correlation data that links user emotions and video quality based on the user emotions of users viewing the video content and the video quality for each scene of the video content. Information processing methods.

7. The system includes a user emotion prediction unit that predicts user emotions for each scene of the video content based on correlation data linking the video quality of each scene of the video content with user emotions. Information processing device.

8. The system further includes a display control unit that controls the display of user emotions for each scene of the predicted video content. The information processing apparatus according to claim 7.

9. The system further includes an extraction unit that extracts emotionally representative scenes based on the user's emotions towards each scene of the video content as predicted. The information processing apparatus according to claim 7.

10. The extraction unit extracts the representative emotional scenes based on the type of user emotion. The information processing apparatus according to claim 9.

11. The extraction unit extracts the emotional representative scene based on the degree of the user's emotion. The information processing apparatus according to claim 9.

12. The extraction unit extracts scenes in which the degree of user emotion exceeds a threshold as the representative emotion scene. The information processing apparatus according to claim 11.

13. The extraction unit extracts the emotional representative scenes based on the statistical value of the overall degree of user emotion in the video content. The information processing apparatus according to claim 11.

14. The aforementioned statistical values ​​include the maximum value, sorting result, mean, or standard deviation. The information processing apparatus according to claim 13.

15. The user emotion prediction unit predicts user emotion for each scene of the video content based on correlation data of predetermined attributes selected from user attribute-specific correlation data. The information processing apparatus according to claim 7.

16. The system further comprises a playback control unit that controls the playback of the video content based on the extracted emotional representative scenes. The information processing apparatus according to claim 9.

17. The system further comprises an editing control unit that controls the editing of the video content based on the extracted emotional representative scenes. The information processing apparatus according to claim 9.

18. The procedure includes predicting user emotions for each scene of a video content based on correlation data that links the video quality for each scene of the video content with user emotions. Information processing methods.

Images