Program interest data extraction device, program interest data extraction method, program interest data extraction program, robot action determination device, robot action determination method, and robot action determination program

Abstract

To provide a program interest data extraction device, capable of extracting a program genre that a user wants to enjoy while sharing a conversation and feeling with the other person and a genre that a user wants to view concentratedly in each user, and controlling an operation pattern and an operation frequency of a television viewing robot on the basis of an extraction result, a method, a program, a robot operation determination device, a method, and a program.SOLUTION: In a feeling expression control device, a program genre classification part calculates a total viewing time in each genre. An expression detection part acquires the number of times of an emotional expression expressed in a face of a user in during the view of a television in each program genre. A gaze data calculation part calculates a time when the user watches a television in the view of the television in each program genre. A program interest level calculation part determines a program genre that the user wants to enjoy while shearing a conversion or feeling and a genre that the user wants to view concentratedly in each user. An operation frequency determination part determines an operation pattern and an operation frequency of a robot during the view of the television in accordance with a determination result of the program interest level calculation part.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technology for a communication robot that watches videos such as television programs together with humans, in which the communication robot finds program content that the user wants to watch with concentration or that the user wants to watch while sharing their emotions with others, and also to a technology for the communication robot to control its movement patterns and movement frequency according to the genre of the program being watched.Unless otherwise specified, the communication robot will be referred to as the robot hereinafter. [Background technology]

[0002] While television can be enjoyed by watching alone, it can also be enjoyed by watching with others, confirming that other people feel the same emotions as you and sharing the enjoyment with others. Patent Document 1 discloses an invention in which robots are included in the "other people" mentioned above.

[0003] The "television viewing robot" described in Patent Document 1 is a communication robot that aims to achieve the same effect as when two humans watch the same television program together by watching television together with a human. This robot extracts keywords from the video, audio, and subtitles of television programs and generates utterances for the robot based on the extracted keywords. This allows the robot to make utterances related to the program, allowing the human and robot to converse while watching television.

[0004] Patent Document 2 discloses a robot control method that aims to keep users engaged in robot use without boring them. It proposes a method for determining the robot's behavior from contact history information when a user touches the robot. Data on the contact area (head, abdomen, etc.), contact type (stroking, hitting, etc.), contact speed, and movement speed included in the contact history information are acquired by sensors, and the robot performs actions based on this data to estimate the user's emotions, allowing users to continue using the robot without boring them.

[0005] Patent Document 3 describes a program that displays a list of content information that a user wants to watch based on their level of interest. User preference information is collected from their TV viewing history, search history on social media, and emotional state while watching TV, and keywords related to programs that interest each user are extracted. This makes it possible to recommend content that interests each user. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2019-185400 A [Patent Document 2] Japanese Patent Application Publication No. 2018-134735 [Patent Document 3] Japanese Patent Application Laid-Open No. 2016-086342 Summary of the Invention [Problem to be solved by the invention]

[0007] One issue with communication robots is that users can grow bored of them over time and stop using them. When using a communication robot to watch TV with a human, the human may be unable to concentrate on the TV if the robot starts moving or talking to them. Conversely, there are times when the human wants to enjoy TV with friends and chat, but the robot remains quiet. This inability of the robot to behave in accordance with the time, place, and occasion (TPO) of TV viewing is thought to be one of the reasons why users do not use the robot for long periods of time.

[0008] In the case of Patent Document 1, the robot extracts keywords from the images and subtitles of the TV program being watched and speaks based on those keywords, making it possible for humans and the robot to have conversations related to the program. However, there is no proposal for the robot's emotional behavior according to the program genre, the user's interests, or whether the user wants to watch alone and concentrate, or whether the user wants to watch with others.

[0009] Patent Document 2 describes a technology that determines the robot's behavior according to the user based on the user's contact history information with the robot. However, it does not determine the robot's behavior when the user is watching television and has not contacted the robot.

[0010] Patent Document 3 relates to a technology for extracting programs that are of high interest to each user based on user information. However, it does not propose a method for finding, among the programs of high interest, programs that users would like to enjoy while sharing conversations and emotions with humans (or communication robots), or programs that users would like to watch with concentration.

[0011] The present invention aims to prevent users from getting bored of a robot and encourage them to continue using it by determining, for each user, the genre of programs that the user wants to enjoy while talking and sharing emotions with the robot, and the genre of programs that the user wants to watch with concentration, based on the television viewing history and the user's facial expression data, and by changing the robot's movement pattern and movement frequency according to the results of this determination, thereby making the robot operate in accordance with the time, place, and occasion of the user when watching television.

[0012] To provide a method for extracting program genres that a user is highly interested in from the history of television programs watched and the user's facial expression data, and extracting for each user program genres that the user wants to enjoy while sharing conversations and feelings with a robot and program genres that the user wants to watch with concentration. [Means for solving the problem]

[0013] The program genres that users want to watch while sharing emotions and having conversations, and the program genres that they want to watch with concentration, differ from user to user. For example, if a user is watching a "drama" program genre that they want to watch with concentration, and there is a robot that expresses emotions and speaks with the same frequency as other program genres, this will disrupt the user's television viewing, causing them to tire of the robot and making it difficult for them to continue using the robot. To solve this problem, we propose a device and method for calculating the two patterns of program genres for each user.

[0014] For each user, for example, based on a week's worth of viewing history and facial expression data, the program genre that showed the most number of facial expressions (number of reactions) recognized during the viewing time and that showed the greatest concentration on TV is set as the program genre that users want to enjoy while sharing conversations and emotions. By setting in this way, it is possible to find times when users can enjoy watching TV with a robot.

[0015] In addition, the program genre that the user is most likely to concentrate on is set as the program genre that the user wants to concentrate on watching. By setting the program genre in this way, the robot can operate less frequently to suit the user, preventing disruption to the user's TV viewing.

[0016] Program genres that have low levels of concentration are set as program genres of which the person is not interested. In such cases, the robot can make utterances related to the program or express emotions as if the person is enjoying watching TV, which can spark the person's interest in the program and lead to new discoveries.

[0017] Using the above method, the robot's movement patterns and frequency can be determined by calculating for each user the "program genres that users want to enjoy by sharing their emotions," "program genres that they want to concentrate on watching," and "program genres that users are least interested in."

[0018] The program interest data extraction device of the present invention is a program interest data extraction device that extracts programs of interest for each user, and includes a program genre classification unit that calculates the total viewing time for each program genre from the viewing history, an expression detection unit that acquires the number of expressions made while watching television for each user, a gaze data calculation unit that acquires the amount of time that each user directed their eyes at the television while watching television, an expression data analysis unit that calculates the number of expressions made and the total gaze time for each program genre, and a program interest level calculation unit that calculates the reaction level and concentration level for each program genre from the number of expressions made and the total gaze time for each user while watching television.

[0019] The robot movement determination device of the present invention is a robot movement determination device that determines the robot's movement for each program genre depending on the user, and determines the robot's movement from the degree of reaction and concentration of the user when watching television, based on the program genre calculated by the program interest data extraction device.

[0020] The program interest data extraction method of the present invention is a program interest data extraction method for extracting programs of interest to each user, and is executed by a computer, comprising: a program genre classification step for calculating the total viewing time for each program genre from the viewing history; an expression detection step for acquiring the number of expressions made while watching television for each user; a gaze data calculation step for acquiring the time spent looking at the television while watching television for each user; an expression data analysis step for calculating the number of expressions made and the total gaze time for each program genre; and a program interest level calculation step for calculating the reaction level and concentration level while watching television for each program genre from the number of expressions made and the total gaze time.

[0021] The robot movement determination method of the present invention is a robot movement determination method that determines the robot's movement for each program genre depending on the user, and a computer executes movement frequency determination that determines the robot's movement from the degree of reaction and concentration based on the user's reaction and concentration when watching television, which correspond to the program genre calculated by the program interest data extraction method.

[0022] A program interest data extraction program according to the present invention is for causing a computer to function as the program interest data extraction device.

[0023] A robot movement determination program according to the present invention is for causing a computer to function as the robot movement determination device. [Effects of the Invention]

[0024] According to the present invention, by determining the robot's behavior depending on the user, it is possible to provide the user with the fun and joy of watching television with multiple people. Furthermore, according to the present invention, the robot's behavior is determined in response to changes in the user's interest in programs, so that the user can continue to use the robot (the user will not tire of the robot). [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a diagram showing an overall block diagram of an embodiment of the present invention; [Figure 2] FIG. 2 is a diagram illustrating the operation of the program data unit 10 according to one embodiment of the present invention. [Figure 3] FIG. 2 is a diagram illustrating the operation of the user data section 20 according to one embodiment of the present invention. [Figure 4] 3A to 3C are diagrams illustrating the operation of a facial expression data analysis unit 31 according to an embodiment of the present invention. [Figure 5] FIG. 2 is a diagram illustrating the operation of a program interest level calculation unit 41 according to an embodiment of the present invention. [Figure 6] FIG. 4 is a diagram illustrating the operation of an operation frequency determination unit 42 according to one embodiment of the present invention. [Figure 7] FIG. 2 illustrates how a user can discover program genres that interest them in one embodiment of the present invention. [Figure 8] 10A and 10B are diagrams illustrating a method for determining the behavior of a robot based on user data according to an embodiment of the present invention. [Figure 9]FIG. 1 is a flowchart illustrating an entire system according to an embodiment of the present invention. [Figure 10] FIG. 2 is a diagram showing a flowchart of the operation frequency data section 40 according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0026] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is an overall block diagram showing one embodiment of the present invention. Reference numeral 100 denotes an emotion expression control device as a robot action determination device incorporated in a robot. Reference numeral 200 denotes a television. Reference numeral 300 denotes a user. The emotion expression control device 100 as a robot movement determination device includes a program data section 10, a user data section 20, a facial expression data section 30, a movement frequency data section 40, a user data DB (database) 70, a movement frequency data DB 80, and a robot movement communication section 90.

[0027] The program data unit 10 includes a viewing history extraction unit 11 and a program genre classification unit 12. The viewing history extraction unit 11 extracts the viewing history of the user (viewing start time, viewing end time, program name, program genre (hereinafter also referred to as "program genre")) for each user. The program genre classification unit 12 calculates the total viewing time for each program genre for one week from the viewing history extracted for each user.

[0028] The user data unit 20 includes a facial expression detection unit 21 and a gaze data calculation unit 22. The facial expression detection unit 21 captures images of the user's face with a camera and calculates the facial expression recognized during the viewing time for each user. The gaze data calculation unit 22 calculates the time during the viewing time that the user directed their gaze at the television (gazing time) based on the direction of the user's eyes captured by the camera.

[0029] The facial expression data unit 30 includes a facial expression data analysis unit 31 and a facial expression data update unit 32. The facial expression data analysis unit 31 uses the data acquired by the program data unit 10 and the user data unit 20 to calculate the total viewing time for a predetermined period (e.g., one week), the number of facial expressions recognized during the viewing time for the predetermined period (one week), and the total time spent looking at the television during the predetermined period (one week), for each user and for each program genre. The facial expression data update unit 32 updates the above data every predetermined period (one week) and stores it in the user data DB 70. Note that in this embodiment, one week is used as an example of the predetermined period unless otherwise specified, but the predetermined period is not limited to one week. The user of this system may set any period in advance.

[0030] The action frequency data unit 40 includes a program interest level calculation unit 41, an action frequency determination unit 42, and an action frequency data update unit 43. The program interest level calculation unit 41 calculates the frequency of the user's facial expressions while watching television (hereinafter referred to as the "reaction level") and the percentage of time during viewing that the user is looking at the television (hereinafter referred to as the "concentration level"). The action frequency determination unit 42 determines, for each user, "a program genre that the user wants to enjoy by sharing their emotions," "a program genre that the user wants to watch with concentration," and "a program genre that the user is less interested in," using the reaction level and concentration level calculated by the program interest level calculation unit 41. Then, it determines the action pattern and action frequency of the robot for each program genre. The action frequency data update unit 43 stores the action patterns and action frequencies of the robot determined for each user and each program genre in the action frequency data DB 80.

[0031] The robot movement communication unit 90 transmits the movement patterns and movement frequencies of the robot determined for each user and each program genre, which are stored in the movement frequency data DB 80, to a control unit (not shown) of the robot. The above has provided an overview of the functional units of the emotional expression control device 100 as a robot movement determination device. Although not shown, as another embodiment, among the functional units of the emotional expression control device 100 as a robot movement determination device, the program data unit 10, the user data unit 20, the facial expression data unit 30, the program interest level calculation unit 41, and the user data DB (database) 70 may be configured as a program interest data extraction device (not shown).

[0032] Furthermore, although not shown, in another embodiment, among the functional units of the emotional expression control device 100, a configuration including the program data unit 10, the user data unit 20, the facial expression data unit 30, and the user data DB 70 may be referred to as a viewing state observation device (not shown), and a configuration including the program interest calculation unit 41, the action frequency determination unit 42, the action frequency data update unit 43, the action frequency data DB 80, and the robot action communication unit 90 may be referred to as a robot action determination device (not shown). In this embodiment, as described above, the emotional expression control device 100 will be described as a robot movement determination device. Next, the functional units of the emotion expression control device 100 as a robot movement determination device will be described in detail with reference to the drawings.

[0033] FIG. 2 is a diagram illustrating the operation of the program data unit 10. As shown in FIG. 2, when the television is turned on and a camera sensor or the like recognizes that the user is in front of the television, the viewing history extraction unit 11 acquires EPG (Electronic Program Guide) information and extracts a viewing history including the names and genres of programs viewed by the user. The program genre classification method extracts program genre information from the acquired viewing history and classifies the program genre based on data of the major genre classifications (0x0 to 0xF). The program genre classification unit 12 extracts information for each program genre from the viewing history acquired by the viewing history extraction unit 11, organizes the information by program genre, and calculates the total viewing time by program genre.

[0034] FIG. 3 is a diagram for explaining the operation of the user data section 20. As shown in FIG. In this embodiment, the facial expression detection unit 21 calculates the number of times a user makes one of four facial expressions while watching television: joy, anger, surprise, and sadness. The facial expressions are detected using a facial expression recognition technology that processes images captured by a camera installed near the television and estimates the type of facial expression. The gaze data calculation unit 22 calculates the time during which the user directs their gaze towards the television (gaze time) within the viewing time. The definition of "within the viewing time" is "the time during which the user is in front of the television while watching a program." The gaze time is detected by using an infrared camera to detect the gaze and calculate the detected time (when the gaze is directed in the direction of the television).

[0035] FIG. 4 is a diagram illustrating the operation of the facial expression data unit 30. As shown in FIG. The facial expression data analysis unit 31 calculates the number of facial expressions (also referred to as the "total number of facial expressions") as the total number of times the user expressed happiness, anger, surprise, and sadness while watching a program in a predetermined period (for example, one week) calculated by the facial expression detection unit 21. Furthermore, based on the data calculated by the gaze data calculation unit 22, the facial expression data analysis unit 31 calculates, for each user and each program genre, the total viewing time for one week, the number of facial expressions recognized within the viewing time for one week, and the total time spent gazing at the television in one week (hereinafter also referred to as the "total gaze time").

[0036] However, program genres (0x2 in Figure 4) with a total viewing time for one week that is less than the threshold (60 minutes) are excluded. The threshold of 60 minutes (= 1 hour) is related to the unit time, which will be described later. In the present invention, the reaction level is calculated by calculating the number of facial expressions per unit time of total viewing time for each program genre. Therefore, program genres with a total viewing time of less than the unit time (1 hour) are excluded from the calculation of the reaction level. In this embodiment, unless otherwise specified, 60 minutes (1 hour) is used as an example of the threshold, but the threshold is not limited to 60 minutes. Users of this system may set it as they wish. The facial expression data update unit 32 updates the user data DB 70 based on the data collected by the facial expression data analysis unit 31 for each user every week.

[0037] FIG. 5 is a diagram illustrating the operation of the program interest level calculation unit 41. As shown in FIG. 5, the program interest level calculation unit 41 calculates the number of facial expressions x per hour for each user and for each program genre based on the data calculated by the facial expression data analysis unit 31, and calculates the average value μ of x across all program genres. The program interest level calculation unit 41 calculates the standard deviation σ based on the calculated number of facial expressions x per hour for each program genre and the average value μ of the number of facial expressions per hour while watching television. Furthermore, the program interest level calculation unit 41 calculates the value (time ratio) z by dividing the total gaze time a for each program genre by the total viewing time n, and calculates the user's concentration level for each program genre based on the value of z. As described above, the program interest level calculation unit 41 calculates, for each user, the user's reaction level while watching TV (the number of facial expressions x per unit time of total viewing time) and concentration level z for each program genre.

[0038] FIG. 6 is a diagram illustrating the operation of the operation frequency determination unit 42. An operation frequency determination unit 42 uses the reaction degree and concentration degree calculated by the program interest degree calculation unit 41 to determine "program genres that users want to enjoy by sharing their emotions," "program genres that users want to watch with concentration," and "program genres that users are less interested in" for each user. The action frequency determination unit 42 classifies the degree of reaction into four levels based on the number of facial expressions x per hour for each program genre, and determines the action pattern and action frequency of the robot.

[0039] Specifically, as shown in Figure 6, for the number of facial expressions per hour for each program genre, x, If μ+σ≦x≦μ+2σ, the reaction level to the program genre is 4, If μ≦x<μ+σ, the reaction level to the program genre is 3, When μ - σ ≤ x < μ, the reaction degree for the program genre is 2. When μ - 2σ ≤ x < μ - σ, the reaction degree for the program genre is 1. Let it be so. In addition, when μ + 2σ < x, the reaction degree for the program genre is 4 plus. When x < μ - 2σ, the reaction degree for the program genre may be set as 1 minus. Although an example of classifying the reaction degree for the program genre into four levels has been shown, it is not limited to four levels. Regarding how many levels to set the reaction degree based on the average value μ and the standard deviation σ, the user of the system may set it in advance.

[0040] Also, in the operation frequency determination unit 42, based on the concentration degree z calculated by the program interest degree calculation unit 41 with 50% as the reference, the program genre with a viewing ratio of 50% or more (looking at the TV for more than half of the time) is defined as the "program genre with a high concentration degree", and the program genre with a viewing ratio less than 50% is defined as the "program genre with a low concentration degree", and they are classified into two categories. Although an example of classifying the concentration degree for the program genre into two levels has been shown, it is not limited to two levels. The user of the system may set it in advance as well.

[0041] [[ID=IS]]Figure 7 is a diagram for explaining a method of discovering a program genre that a user is interested in. Based on the frequency of the above reaction degree and the frequency of the concentration degree, determine the program genre that the user is interested in or not interested in. When the reaction degree is (high) and the concentration degree is (high), it is a program genre that the user is interested in and wants to share the feelings and enjoy with others. Generally, music programs and variety shows are considered to be applicable. Also, when the reaction degree is (low) and the concentration degree is (high), it is set as a program genre that the user is interested in and wants to watch concentratedly. Generally, dramas and movies that the user gets into the story and takes the position of the characters are considered to be applicable. And when the concentration degree is (low), it is set as the "program genre that the user is not interested in". Generally, news and reporting programs with constantly changing themes, and sports programs without exciting scenes are considered to be applicable. Of course, the layout shown in Figure 7 is just one example. For a user who likes Go or Shogi, the "Hobbies" program genre might be located at the far right of Figure 7.

[0042] FIG. 8 is a diagram showing a method for determining the robot's movements based on user data by the movement frequency determination unit 42. The movement frequency determination unit 42 determines the movement pattern and movement frequency of the robot based on the user data indicating the degree of interest in the program genre. The action frequency data updating unit 43 adds to and updates the action frequency data DB 80 data relating to the action patterns and action frequencies for each program genre, which data has been determined by the action frequency determining unit 42 for each user.

[0043] Furthermore, the robot movement communication unit 90 transmits the movement pattern and movement frequency of the robot determined by the movement frequency determination unit 42 to a robot control unit (not shown). In this way, when the user is watching a program genre of their choice, the robot (not shown) can control its movement based on the movement pattern and movement frequency that matches the user's time, place, and occasion when watching television. For example, when a program genre with a high reaction rate (4 on a 4-point scale) is watched, the robot performs emotional movements and speech movements in accordance with the number of facial expressions made by the user, and performs movements to enjoy the program together with the user. Specific examples of movements include emotional movements such as laughing and speech movements such as empathetic responses.

[0044] Furthermore, when watching a program genre with a relatively high reaction rate (3 on a 4-point scale) or a relatively low reaction rate (2 on a 4-point scale), the robot will perform speech actions related to the program to make the user more interested in the program. Specific examples of its actions include generating speech based on keywords extracted from television information and engaging in a dialogue with the user.

[0045] When watching a program genre with a low reaction level (1 on a 4-point scale), the robot will perform emotional movements to make the user look at the TV or become interested in it, as if they are enjoying the program. A specific example of this behavior is the robot performing emotional movements such as happiness, as if it is enjoying watching the program. The functional units of the emotion expression control device 100 as a robot operation determination device have been described above. As mentioned above, a configuration including the program data unit 10, the user data unit 20, the facial expression data unit 30, the program interest level calculation unit 41, and the user data DB (database) 70 may be configured as a program interest data extraction device (not shown).

[0046] Fig. 9 is a flowchart of the entire system according to one embodiment of the present invention, and Fig. 10 is a flowchart of the process from discovering a program of interest to determining the robot's movement pattern and frequency of movement. Steps S905 and S906 in Fig. 9 are executed by the action frequency data unit 40 and correspond to steps S1001 to S1006 in Fig. 10. Steps S1004 and S1005 in Fig. 10 are executed by the action frequency determination unit 42.

[0047] As shown in FIG. 9, in step S901, the program data unit 10 calculates and obtains the total viewing time (total viewing time) of all programs belonging to each program genre for a predetermined period (one week) from the history of television programs viewed by the user.

[0048] In step S902, the facial expression detection unit 21 of the user data unit 20 records the number of times that the user's face shows happiness, anger, surprise, and sadness while watching television, along with the program genre. Also, in step S902, the gaze data calculation section 22 of the user data section 20 records the time during which the user directs his / her gaze towards the television while watching television as gaze time together with the program genre.

[0049] In step S903, the facial expression data analysis unit 31 of the facial expression data unit 30 totals the number of times that the facial expression detection unit 21 has recorded the user's face, including the number of times that the user has expressed happiness, anger, surprise, and sadness. In step S903, the facial expression data analysis unit 31 of the facial expression data unit 30 tally up the total number of facial expressions for a predetermined period (one week) by program genre. In step S903, the facial expression data analysis unit 31 of the facial expression data unit 30 calculates the total gaze time by tallying the gaze times calculated by the gaze data calculation unit 22 for a predetermined period (one week) by program genre. Also, in step S903, the facial expression data analysis unit 31 of the facial expression data unit 30 compiles the counted number of facial expressions, total gazing time, and total viewing time for a predetermined period (one week) acquired by the program data unit 10, and creates a table by program genre.

[0050] In step S904, the facial expression data update unit 32 of the facial expression data unit 30 updates the user data DB70 by adding the data compiled by the facial expression data analysis unit 31 by user and by program genre every predetermined period (one week) to the user data DB70.

[0051] In step S905, the behavior frequency data unit 40 calculates the robot's behavior pattern and behavior frequency based on the number of facial expressions, total gaze time, and total viewing time accumulated in the user data DB 70 by program genre for a predetermined period (one week).

[0052] In step S906, the movement frequency data update unit 43 of the movement frequency data unit 40 stores the movement pattern and movement frequency of the robot calculated in step S905 in the movement frequency data DB 80.

[0053] In step S907, the robot operation communication unit 90 transmits the operation pattern and operation frequency of the robot stored in the operation frequency data DB 80 to a control unit (not shown) of the robot. The processing flow of the entire system according to one embodiment of the present invention has been described above.

[0054] As shown in FIG. 10, in step S1001, the program interest level calculation unit 41 of the behavior frequency data unit 40 acquires the total viewing time, number of facial expressions, and total gazing time for each program genre for one week for each user from the user data DB 70.

[0055] In step S1002, the program interest calculation unit 41 of the behavior frequency data unit 40 calculates the number of facial expressions per hour x based on the total viewing time and the number of facial expressions, and calculates the frequency (reaction level) of the number of facial expressions while watching television by program genre based on the average value μ and standard deviation σ.

[0056] In step S1003, the program interest calculation unit 41 of the behavior frequency data unit 40 calculates the proportion of time (degree of concentration) that the viewer directs their gaze toward the television while watching television by program genre, based on the total viewing time and total gazing time.

[0057] In step S1004, the program interest calculation unit 41 of the action frequency data unit 40 determines, for each user, a program genre that the user is interested in and wants to enjoy, a program genre that the user is interested in and wants to watch with concentration, and a program genre that the user is not interested in, based on the reaction degree and the concentration degree.

[0058] In step S1005, the operation frequency determination unit 42 of the operation frequency data unit 40 determines the operation pattern and operation frequency of the robot based on the program genres that the user is interested in and wants to enjoy, the program genres that the user is interested in and wants to watch with concentration, and the program genres that the user is not interested in, which are determined for each user.

[0059] In step S1006, the action frequency data update unit 43 of the action frequency data unit 40 updates the action frequency data DB80 by adding the data on the robot's action pattern and action frequency determined in step S1005 to the action frequency data DB80. The above has described the processing flow for adding and updating the action frequency data DB 80 for each program genre of a user based on the user data DB 70.

[0060] [Variation 1] In the embodiment of the present invention, it is assumed that a user is watching television alone, but the present invention can also be applied to a case where a plurality of users are watching the same program together. If there are multiple people watching TV, it is possible to extract program genres that are of interest or disinterest to each group, rather than by user, and determine the robot's movement patterns and frequency of movement. That is, the number of facial expressions and total gaze time of each user watching TV are detected, and the reaction level and concentration level are added up and averaged to determine the robot's behavior. Alternatively, if there are users in a group who are interested in the program currently being watched and users who are not, there are methods such as making the robot behave in accordance with the users who are interested in the program, or making program-related utterances to users who are not interested in the program, giving them an opportunity to become interested (which has the effect of getting more people interested in the program and livening up the TV viewing environment).

[0061] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. In the present embodiment, the emotional expression control device 100 as a robot action determination device is connected to a robot (not shown) so as to be able to communicate with the robot. However, as described above, a program interest data extraction device may also be provided. Furthermore, as mentioned above, the system configuration may be such that a configuration including the program data unit 10, the user data unit 20, the facial expression data unit 30, and the user data DB 70 is a viewing state observation device (not shown), and a configuration including the program interest level calculation unit 41, the action frequency determination unit 42, the action frequency data update unit 43, the action frequency data DB 80, and the robot action communication unit 90 is a robot action determination device (not shown).

[0062] Furthermore, a robot (not shown) may be provided with the emotional expression control device 100 as a robot movement determination device. The program interest data extraction device and the robot movement determination device may each be integrated into a single device. The functional units of the emotional expression control device 100 may be configured as a distributed system. For example, some of the functional units of the emotional expression control device 100 may be implemented on a cloud server, for example. Furthermore, the effects described in the above-described embodiments are merely a list of the most favorable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments.

[0063] In this embodiment, the configuration and operation of the emotion expression control device 100 have been mainly described, but the present invention is not limited to this and may be configured as a method or program having each component for causing a robot to express emotions.

[0064] Furthermore, the functions of the emotional expression control device 100 may be realized by recording a program on a computer-readable recording medium, and reading and executing the program recorded on this recording medium into a computer system.

[0065] The term "computer system" here includes hardware such as the OS and peripheral devices. Additionally, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems.

[0066] Furthermore, the term "computer-readable recording medium" may include a medium that dynamically stores a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, or a medium that stores a program for a fixed period of time, such as a volatile memory within a computer system that serves as a server or client in such a case. Furthermore, the program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system. [Explanation of symbols]

[0067] 100 Emotion Expression Control Device 200 TV 300 users 10 Program Data Section 11 Viewing history extraction unit 12 Program genre classification section 20 User Data Section 21 Facial Expression Detection Unit 22 Gaze data calculation unit 30 Facial Expression Data Section 31 Facial Expression Data Analysis Department 32 Facial expression data update unit 40 Operation frequency data section 41 Program Interest Calculation Unit 42 Operation frequency determination unit 43 Operation frequency data update unit 70 User Data DB 80 Operation Frequency Data DB 90 Robot movement communication unit

Claims

1. A program interest data extraction device that extracts programs of interest to each user, a program genre classification unit that calculates a total viewing time for each program genre from the viewing history; a facial expression detection unit that acquires the number of facial expressions made by each user while watching television; a gaze data calculation unit for acquiring, for each user, the time spent looking at the television while watching the television; a facial expression data analysis unit that calculates the number of facial expressions and the total gaze time for each program genre; The program interest data extraction device includes a program interest calculation unit that calculates the reaction level and concentration level when watching television for each program genre based on the number of facial expressions and the total viewing time.

2. A robot action determination device that determines robot actions for each program genre according to a user, Based on the reaction level and concentration level of the user when watching television, corresponding to the program genre calculated by the program interest data extraction device according to claim 1, A robot action determination device that determines the action of the robot based on the degree of reaction and the degree of concentration.

3. A program interest data extraction method for extracting programs of interest to each user, a program genre classification step for calculating a total viewing time by program genre from the viewing history; An expression detection step of acquiring the number of expressions made by each user while watching television; A gaze data calculation step for acquiring the time spent looking at the television while watching television for each user; A facial expression data analysis step for calculating the number of facial expressions and total gaze time by program genre; a program interest level calculation step of calculating the reaction level and concentration level during television viewing for each program genre from the number of facial expressions and the total viewing time, the program interest level extraction method being executed by a computer.

4. A robot action determination method for determining a robot action for each program genre according to a user, Based on the reaction level and concentration level of the user when watching television, corresponding to the program genre calculated by the program interest data extraction method according to claim 3, A robot action determination method in which a computer executes action frequency determination to determine the robot's action based on the reaction degree and the concentration degree.

5. 2. A program interest data extraction program for causing a computer to function as the program interest data extraction device according to claim 1.

6. A robot movement determination program for causing a computer to function as the robot movement determination device according to claim 2.

7. A robot action determination device that determines robot actions for each program genre according to a user, a program interest calculation unit that calculates a reaction level and a concentration level while watching television for each program genre based on the number of facial expressions and the total gaze time while watching television; A robot movement determination device comprising: a movement frequency determination unit that determines the movement of the robot based on the degree of reaction and the degree of concentration.

8. A robot action determination method for determining a robot action for each program genre according to a user, a program interest calculation step for calculating the reaction level and concentration level during television viewing for each program genre based on the number of facial expressions and total gaze time during television viewing; and a movement frequency determination step of determining a movement of the robot based on the degree of reaction and the degree of concentration.

9. A robot movement determination program for causing a computer to function as the robot movement determination device according to claim 7.

Images