Information processing apparatus, learning apparatus, content providing system, information processing method, and content providing method

Abstract

To provide new contents.SOLUTION: An information processing apparatus includes an acquisition unit that acquires vital data that is a measurement result measured by a vital sensor from a person who may appear in displayable content, an emotion estimation unit that estimates emotion data indicating an emotion of the person on the basis of the acquired vital data, and an output data generation unit that generates output data that can be displayed together with the content and that is based on the estimated emotion data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a learning device, a content providing system, an information processing method, and a content providing method. [Background technology]

[0002] Various content such as sports programs, discussion programs, quiz programs, etc. are provided via television broadcasting, internet distribution, etc. Program producers may enhance the content by superimposing captions, etc., corresponding to the broadcast content, on the display screen of the content. For example, in sports programs, captions are used to display characters and figures explaining the progress of the game, while in debate programs, captions are used to display the affiliations, names, careers, etc. of the participants, and subtitles are used to display text corresponding to the content of their remarks. A video synthesizer that displays subtitles over the main video has also been provided (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-170822 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the content of the captions superimposed on the content may be sentences that the program producers have thought up based on the content, or may be text that has been written by the performers. Therefore, while it is possible to superimpose on the display screen what the performers look like from a third-party perspective or what the performers have said, it would be desirable to be able to provide new content that takes into account other points.

[0005] The present invention has been made in consideration of the above circumstances, and its purpose is to provide an information processing device, a learning device, a content provision system, an information processing method, and a content provision method that are capable of providing new content. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, one aspect of the present invention includes an acquisition unit that acquires vital data, which is a measurement result measured by a vital sensor from a person who may appear in displayable content; an emotion estimation unit that estimates emotion data that represents the emotion of the person based on the acquired vital data; and an output data generation unit that generates output data that can be displayed together with the content and is based on the estimated emotion data.

[0007] Another aspect of the present invention is a learning device that learns the relationship between emotions of characters that may appear in displayable content and output data that may be displayed together with the content.

[0008] Another aspect of the present invention is a content providing system including the above-mentioned information processing device and a content output device, wherein the information processing device transmits output data to the content output device based on emotion data estimated based on vital data acquired when the person appears in the content, and the content output device has a receiving unit that receives the output data transmitted from the information processing device, and a display unit that displays the content from a content distribution device that distributes the content, and displays the output data transmitted from the information processing device together with the content.

[0009] Another aspect of the present invention is an information processing method executed by a computer, the information processing method including: acquiring vital data, which is a measurement result measured by a vital sensor, from a person who may appear in displayable content; estimating emotional data representing an emotion of the person based on the acquired vital data; and generating output data based on the estimated emotional data, which is output data that can be displayed together with the content.

[0010] Another aspect of the present invention is a content providing method executed in a content providing system including an information processing device and a content output device, wherein the information processing device acquires vital data, which is a measurement result measured by a vital sensor, from a person who may appear in displayable content, estimates emotional data representing an emotion of the person based on the acquired vital data, generates output data based on the estimated emotional data, which is output data that can be displayed together with the content, and when the output data based on the emotional data estimated based on vital data acquired when the person appears in the content is generated, transmits the generated output data to the content output device, and the content output device receives the output data transmitted from the information processing device, receives the content from a content distribution device that distributes the content and displays it, and displays the output data transmitted from the information processing device together with the content. [Effects of the Invention]

[0011] As described above, according to the present invention, new content can be provided. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a schematic block diagram showing the configuration of a content providing system S according to an embodiment of the present invention. [Figure 2] FIG. 2 is a functional block diagram showing an outline of functions of an information processing device 40. [Figure 3] FIG. 10 is a diagram illustrating an example of behavior data. [Figure 4] FIG. 2 is a schematic functional block diagram showing the functions of a display device 70. [Figure 5] 10 is a flowchart illustrating a flow of generating and using emotion estimation data. [Figure 6] FIG. 10 is a diagram showing emotion data obtained based on a plurality of indices from vital data. [Figure 7] 1 is a conceptual diagram illustrating the flow of operations of the content providing system S. FIG. [Figure 8] 10 is a flowchart illustrating the operation of the information processing device 40 in the execution phase. [Figure 9] FIG. 1 is a conceptual diagram illustrating a flow when a baseball game is distributed as content. [Figure 10] 10 is a diagram showing an example of a display screen displayed on a display device 70. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0013] A content providing system according to an embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a schematic block diagram showing the configuration of a content providing system S according to an embodiment of the present invention. The content providing system S includes a plurality of terminal devices 10 (terminal device 10a, terminal device 10b, ... terminal device 10n), a plurality of vital sensors 15 (vital sensor 15a, vital sensor 15b, ... vital sensor 15n), a content server 20, a learning device 30, an information processing device 40, a broadcasting device 50, a television set 60, a display device 70, and a network NW.

[0014] The multiple terminal devices 10 (terminal device 10a, terminal device 10b, ... terminal device 10n) are used by different users, and may be, for example, any of smartphones, tablets, personal computers, etc. Each of the multiple terminal devices 10 is communicably connected to a network NW, and communicates with other devices connected to the network NW.

[0015] The multiple vital sensors 15 (vital sensor 15a, vital sensor 15b, ... vital sensor 15n) are used by different users, and perform biological sensing of the pulse, breathing, etc. from the movements of the body surface of the user, and generate vital data representing the sensing results. The vital data is data in which values ​​representing the measurement results are arranged in chronological order.

[0016] Vital sensor 15 may be a contact type that senses the user's biological activity by contacting a part of the user's body, or a non-contact type that senses the user's biological activity without contacting the user. If vital sensor 15 is a non-contact type, it senses the user's biological activity by, for example, irradiating the user with microwaves using a Doppler sensor and detecting the biological activity based on the reflected waves. The non-contact type vital sensor 15 can measure the user from any distance, but a model that can perform measurements from a distance of, for example, about 10 cm to several meters may be used. When the non-contact vital sensor is installed indoors, for example, it may be attached to a wall, ceiling, fixture, furniture, stand, etc., and may measure the person being measured.

[0017] The contact-type vital sensor may be, for example, an existing electronic device that is built into a wristwatch and sold for sale. Such wristwatch-type vital sensors are widely available, and the number of users who already own them is increasing. Furthermore, even if the contact-type vital sensor is not built into a wristwatch, it may be a wearable type that can be worn on the wrist, arm, chest, abdomen, or the like. When the non-contact vital sensor is installed outdoors, for example, it may be attached to a wall, fence, roof, eaves, mounting base, etc., and used to measure the person being measured. Non-contact vital signs sensors are increasingly being installed in homes and businesses, creating an environment where they can be easily used. In this way, at least one of a contact type and a non-contact type may be used as the vital sensor.

[0018] The terminal device 10a and the vital sensor 15a may be used, for example, by a first user, the terminal device 10b and the vital sensor 15b may be used, for example, by a second user, and the terminal device 10n and the vital sensor 15n may be used, for example, by an Nth user. The number of combinations of terminal devices 10 and vital sensors 15 may be the same as the number of users, and may be, for example, two or more. Furthermore, one set of terminal device 10 and vital sensor 15 may be used by multiple users in rotation. Vital sensor 15a is connected to terminal device 10a via wireless communication by performing a pairing process, and vital data, which is the measurement result, may be transmitted to terminal device 10a wirelessly or via a wired connection. Similarly, vital sensor 15b and terminal device 10b, and vital sensor 15n and terminal device 10n are connected to each other wirelessly or via a wired connection so that they can communicate with each other.

[0019] The contact-type vital sensor 15 may be attached to a person to be measured, and the terminal device 10 may be placed near the person wearing the sensor, and the vital data may be transmitted to the information processing device 40. The non-contact vital sensor 15 may acquire vital data of a person to be measured from a position away from the person, and transmit the vital data to the information processing device 40 from a terminal device 10 installed near the vital sensor 15. When the non-contact vital sensor 15 is used, the terminal device 10 that is the communication partner may be one that is personally owned by the user, or may be one that is installed in a facility where the non-contact vital sensor 15 is installed.

[0020] The vital sensor 15 may be communicably connected to the network NW without going through the terminal device 10, and may be configured to transmit vital data to other devices connected to the network NW (e.g., the learning device 30, the information processing device 40).

[0021] The content server 20 distributes content, which may include people, such as video data, and may also include still images. The content may be, for example, live or recorded. More specifically, the content relates to an action based on either a competition or a game, and such content may be video data of a competition played by a contestant, video data of a game played by a contestant, or video data of a program in which a person appears. The content may feature one or more people. When multiple people appear, the content may be a video of a competition or game in which multiple competitors compete against each other.

[0022] The competition may be, for example, any of baseball, soccer, tennis, table tennis, billiards, darts, bowling, etc. The competition may be an individual competition or a team competition in which players can participate as a team. The game may be, for example, any of shogi, go, karuta, e-sports, etc. The game may be an individual game or a team game in which players can participate as a team. The program may be, for example, any of variety shows, quiz shows, news shows, discussion shows, travel shows, and the like.

[0023] The learning device 30 performs learning using training data and generates a trained model. The learning device 30 uses as training data the relationship between the vital data of the user while performing the behavior specified as the measurement target (such as a sport, game, or TV appearance) and the emotions of the user while performing the behavior as determined by a questionnaire given to the user, and learns the relationship between the vital data and emotions, thereby generating an emotion estimation model (trained model) that estimates emotions from the vital data.

[0024] In addition, the learning device 30 generates an output data generation model (trained model) that generates output data by learning the relationship between the emotions of people who may appear in the displayable content and the output data that can be displayed together with the content.

[0025] The learning method performed by the learning device 30 may be machine learning or deep learning. The learning device 30 is communicably connected to a network NW and transmits the generated trained model to the information processing device 40 via the network NW.

[0026] The information processing device 40 is communicably connected to the terminal device 10, the content server 20, the learning device 30, the broadcasting device 50, etc. via the network NW. The information processing device 40 may also be communicably connected to the vital sensor 15 via the network NW. The information processing device 40 may be a physical server or a cloud server provided by a cloud computing service.

[0027] The broadcasting device 50 transmits broadcast content to viewer devices (such as a television receiver 60 and a display device 70). The broadcasting device 50 may be a device that broadcasts television content using a television communication method, or may be a device that broadcasts content by distributing it over a network. There may be a plurality of broadcasting devices 50, for example, one for each broadcasting station.

[0028] The television receiver 60 receives the content broadcast by the broadcasting device 50 and outputs images and sounds according to the content. The television receiver 60 may be installed in the user's home, workplace, store, restaurant, public facility, or the like.

[0029] The display device 70 has a function of outputting video and may also have a function of outputting sound. The display device 70 displays content and output data transmitted from the information processing device 40 on a liquid crystal display panel or the like, for example. The display device 70 also has a speaker and can output sound based on a sound signal included in the content from the speaker. Furthermore, if the output data includes a sound signal, the display device 70 may output the sound signal, which is the output data, from the speaker. The display device 70 may also be a projector. Display device 70 may be installed in any of the user's home, workplace, store, restaurant, public facility, etc. Display device 70 may also be a large display installed in at least one of a station, commercial facility, stadium (baseball field, soccer field, athletics stadium, game venue, gymnasium), etc. For example, when display device 70 is installed in a baseball stadium, it may be a large image display device installed near the scoreboard and visible to spectators.

[0030] The network NW may be the Internet, a LAN (Local Area Network), or a combination of these. The network NW may also include a communication network for communication using a television communication method.

[0031] There may be one or more of each of the broadcasting device 50, the television receiver 60, and the display device 70.

[0032] FIG. 2 is a functional block diagram showing an outline of the functions of the information processing device 40. As shown in FIG. The information processing device 40 includes a communication unit 401 , a storage unit 402 , a control unit 403 , an acquisition unit 404 , a feeling estimation unit 405 , an output data generation unit 407 , and an output data provision unit 408 . The communication unit 401 communicates with external devices via the network NW.

[0033] The storage unit 402 stores various types of data. The memory unit 402 is configured by a storage medium, such as a hard disk drive (HDD), flash memory, electrically erasable programmable read-only memory (EEPROM), random access read / write memory (RAM), read-only memory (ROM), or any combination of these storage media. The storage unit 402 may be, for example, a nonvolatile memory.

[0034] For example, the storage unit 402 stores an emotion estimation data storage unit 4021 and a behavior storage unit 4022. The emotion estimation data storage unit 4021 stores emotion estimation data that indicates the correspondence between vital data and emotions. The emotion estimation data may be an emotion estimation model, which will be described later. The behavior storage unit 4022 stores behavior data in which emotion data is associated with state data that indicates the results of the behavior. Here, the storage unit 402 may store the output data generation model generated by the learning device 30.

[0035] FIG. 3 is a diagram illustrating an example of behavior data. The behavior data shown in this figure is data in which a person ID, emotion data, and state data are associated with each other. The person ID is identification information that identifies a person who may appear in the content. The status data represents the results of an action. For example, the status data may represent the results of either a competition or a game, or may represent the results of a match. The status data in FIG. 3 is an example of status data when the content is baseball and the type is a pitcher, and includes items such as hits, walks, hit by pitch, and strikeouts. For example, a person ID "A111" is associated with emotion data "motivation A1-A2%" and status data (a hit is "C1", a walk is "C2", a hit by pitch is "C3", and a strikeout is "C4"). Here, for a pitcher assigned person ID "A111", if the estimated emotion data is within the range of "motivation A1-A2%," when facing a batter, the match results are as follows: a hit is "C1", a walk is "C2", a hit by pitch is "C3", a strikeout is "C4", and so on. Here, for each emotion data of each person, past performance up to a certain period of time is tallied, and the performance is stored in advance as status data in the behavior memory unit 4022.

[0036] The acquisition unit 404 acquires vital data, which is the measurement results measured by a vital sensor, from a person who may appear in the displayable content. The acquisition unit 404 may acquire time-series vital data measured during a period in which a person may appear in content and an activity (such as a sport, game, or program) is taking place.

[0037] The emotion estimation unit 405 estimates emotion data representing the emotion of a person who may appear in the content, based on the vital data of the person. For example, the emotion estimation unit 405 may estimate the emotion data by scoring based on the transition of time-series data of the vital data that belongs to a certain past range from the time when the emotion is estimated. The emotion estimation unit 405 may estimate emotion data of a person based on vital data acquired while the person is performing an action. The emotion estimation unit 405 may estimate emotion data of each of the targeted individuals based on vital data acquired for at least two individuals among a plurality of individuals belonging to the same team.

[0038] The output data generation unit 407 generates output data that can be displayed together with the content and is based on the estimated emotion data. Furthermore, output data generation section 407 may read out state data corresponding to estimated emotion data, and generate output data based on the read out state data. Furthermore, output data generation unit 407 may read out the results, which are state data corresponding to the estimated emotion data, and generate output data based on the read out results. The output data generation unit 407 may read out the match results corresponding to the combination of the emotion data obtained from the first person and the emotion data obtained from the second person from the behavior memory unit 4022, and generate output data based on the read out match results. Furthermore, the output data generation unit 407 may estimate the psychological state of the team to which the target person belongs based on the emotional data of each of the target people, and generate output data based on the estimated psychological state.

[0039] The output data providing unit 408 transmits output data based on emotion data estimated based on vital data acquired when a person appears in the content to an external device via the communication unit 401. The external device may be a content output device capable of outputting content, or a device capable of transmitting data to the content output device. The content output device may be, for example, any of the terminal device 10, the television set 60, and the display device 70. When the output data is transmitted to the content output device, the content output device can display the output data together with the content on a display screen.

[0040] The device capable of transmitting data to the content output device may be, for example, either the content server 20 or the broadcast device 50. When output data is transmitted to the content server 20 or the broadcast device 50, the content server 20 or the broadcast device 50 can transmit the output data together with the content. For example, the content server 20 or the broadcast device 50 can add the output data to the content and transmit it. In this case, the content server 20 may display the output data by overlapping it with the display area in which the video data, which is the content, is displayed, or may display the output data in a display area adjacent to the display area of ​​the video data so that the content does not overlap. The broadcasting device 50 may broadcast the content and the output data by data broadcasting. Furthermore, the process of overlaying output data on content may be performed by any one of the terminal device 10, the television receiver 60, and the display device 70. For example, the content server 20 may distribute the content, the broadcasting device 50 may broadcast the content, and the information processing device 40 may distribute output data corresponding to these contents. Then, for example, the display device 70 may display the output data received from the information processing device 40 together with the content received from the content server 20.

[0041] The communication unit 401, the control unit 403, the acquisition unit 404, the emotion estimation unit 405, the output data generation unit 407, and the output data provision unit 408 of the information processing device 40 may be configured by a processing device such as a CPU (Central Processing Unit) or a dedicated electronic circuit.

[0042] FIG. 4 is a schematic functional block diagram showing the functions of the display device 70. The display device 70 includes a communication unit 701 , a storage unit 702 , a control unit 703 , an input unit 704 , and an output unit 705 . The communication unit 701 communicates with external devices via the network NW. The communication unit 701 also has a function of receiving output data transmitted from the information processing device 40 and a function of receiving content distributed from the content server 20. The control unit 703 controls each unit of the display device 70. For example, the control unit 703 causes the output unit 705 to output content, output data, and the like. The input unit 704 receives operation input from the user. The input unit 704 may be, for example, at least one of a keyboard, a mouse, a touch panel, an operation button, and the like. When the communication unit 701 receives content from a content distribution device (e.g., content server 20) that distributes the content, the output unit 705 displays the received content together with the output data transmitted from the information processing device 40. The output unit 705 may be a liquid crystal display panel. Furthermore, the output unit 705 may not only display images but also have a speaker and output sounds included in the content or the output data.

[0043] Here, the functions of the display device 70 have been described, but the functions of the communication unit 701, memory unit 702, control unit 703, input unit 704, and output unit 705 may be implemented in a content output device having the function of outputting content, and may be implemented in, for example, the terminal device 10 or the television receiver 60.

[0044] FIG. 5 is a flow diagram illustrating the flow of generating and using emotion estimation data. The emotion estimation data is data that represents the correspondence between vital data and emotions. A user U1 who can provide vital data used to generate the emotion estimation data is targeted. The user U1 is asked to perform a behavior designated as a measurement target. The vital data is acquired by having the user U1 perform a biological activity during the period in which the behavior is being performed using a vital sensor 15 that measures the user U1. This behavior may be related to the content described above. For example, if the content is a broadcast program broadcasting a baseball game, the behavior may be a baseball game. If the content is a broadcast program broadcasting the progress of a shogi game, the behavior may be playing shogi. Furthermore, if the content is a variety show, the behavior may be the behavior of a cast member in the variety show, or the behavior of a person who may appear in the variety show. The behavior of a person who may appear in the variety show may be behavior during an appearance on the variety show, or may be behavior during a period other than the period in which the variety show is being filmed (behavior before filming the program, behavior after filming the program, behavior during rehearsals).

[0045] User U1 may be a person who may appear in the content. For example, if the content is a baseball game, user U1 may be a player who may participate in the baseball game. Also, if the content is a shogi game, user U1 may be a player in the shogi game. Furthermore, a questionnaire about the emotions felt while performing the behavior is conducted on the person who performed the behavior, and the results of the questionnaire are input from the terminal device 10 or a separately provided management device, etc. Then, the vital data of the user U1 while performing the behavior specified as the measurement target and the emotions resulting from the questionnaire are associated and stored as emotion estimation data in the emotion estimation data storage unit 4021. Here, the user to be measured may be a single person, user U1, or measurements may be taken of multiple different users.

[0046] Furthermore, using as training data the relationship between the vital data of user U1 while performing the behavior specified as the measurement target and the emotions of user U1 ascertained through a questionnaire, the learning device 30 learns the relationship between the vital data and emotions, thereby generating a trained model for estimating emotions from the vital data, and transmits the trained model to the information processing device 40. The control unit 403 of the information processing device 40 may receive the emotion estimation model and store it in the emotion estimation data storage unit 4021. In this way, by using the emotion estimation model, the information processing device 40 can estimate emotion data from trends in changes in vital data such as heart rate and pulse rate.

[0047] This makes it possible to generate emotion estimation data or an emotion estimation model that takes into account the causal relationship between vital data and emotions by using the correspondence between vital data and actual emotions based on a questionnaire. Alternatively, an existing estimation engine that estimates emotions from vital data may be used.

[0048] After the emotion estimation data is stored in the emotion estimation data storage unit 4021 in this way, when vital data of a user U2 (e.g., a person appearing in the content) who is performing the behavior that is the subject of the content is obtained, the emotion estimation unit 405 reads out the emotion data corresponding to this vital data by referring to the emotion estimation data storage unit 4021, thereby obtaining the emotion data corresponding to the vital data. Furthermore, the emotion estimation unit 405 may obtain emotion data by inputting vital data into an emotion estimation model.

[0049] Here, the index of emotion data obtained from vital data may be one or more. For example, the emotional data may include a value representing the degree of at least one of the following indicators obtained by scoring the vital data: "excitement," "relaxation," "fatigue," "tension," "concentration," "motivation," etc. 6 is a diagram showing emotion data obtained from vital data based on multiple indices. As an example, the diagram shows a case where a waveform 801 indicating a first index "excitement," a waveform 802 indicating a second index "fatigue," and a waveform 803 indicating a third index "motivation" are obtained from the vital data of user Ua.

[0050] The scoring process performed by the emotion estimation unit 405 may be, for example, based on the frequency with which emotions corresponding to the indexes of the evaluation target are expressed in an evaluation interval (e.g., a range of a certain period from the time of measurement) among the users to be evaluated.

[0051] More specifically, the degree of emotion corresponding to the index of the evaluation target may be determined by evaluating the emotion based on the cumulative time during which the emotion, which indicates good or bad, is expressed. Alternatively, the degree may be determined by making an evaluation based on the integral value (area) of the region in which the emotion, which indicates good or bad, is expressed. Such scoring may be performed by the emotion estimation unit 405 or the output data generation unit 407.

[0052] In this way, the emotion estimation unit 405 may obtain emotion data for multiple indices from the vital data. In this case, the output data generation unit 407 may generate output data according to one index, or may generate output data according to the degree of relationship between each of the multiple indices.

[0053] Next, the operation of the content providing system S will be described. FIG. 7 is a conceptual diagram illustrating the flow of operations of the content providing system S. The operation of the content providing system S can be roughly divided into a "preparation phase" and an "execution phase."

[0054] Preparation Phase The preparation phase is a phase in which emotion estimation data (or emotion estimation model) is generated and stored in emotion estimation data storage unit 4021. Here, a plurality of users (for example, people who may appear in the content) are targeted, and while the users are engaged in an action (sport, game, appearance on a program, program rehearsal, etc.) that has been determined as the measurement target (step S1), vital data is measured for each user using vital sensor 15. By performing measurements using vital sensor 15, vital data that is time-series data that changes according to emotions can be obtained.

[0055] When vital data is measured by the vital sensor 15, the terminal device 10 acquires the vital data from the vital sensor 15. The terminal device 10 asks the user to enter answers to a questionnaire about emotions during the measurement period by the vital sensor 15 along with the acquired vital data, and transmits the vital data and the emotions resulting from the questionnaire to the learning device 30. The learning device 30 acquires data associating such vital data with emotions from each of the multiple users (step S2) and stores the data in a storage device within the learning device 30. Here, the learning device 30 may classify and store the vital data and the emotions, which are the questionnaire results, received from the terminal device 10 by type of behavior. The type of behavior may be, for example, the type of behavior performed while measurement by the vital sensor 15 was being performed. For example, the type of behavior may be the type of athletic event being performed as content, the type of game, or the category of a program. The type of behavior may also be a further subdivision of the athletic event. For example, if the athletic event is baseball, the type of behavior may be a baseball pitcher, batter, manager, runner on base, infielder, outfielder, etc. If the behavior is a program, the type may be a variety show, news program, discussion program, etc. Here, the terminal device 10 is asked to register in advance, as user data, the relationship between terminal identification information individually assigned to the terminal device 10 and the type of behavior in the learning device 30. As a result, when the learning device 30 receives vital data and survey results from the terminal device 10, it can have the terminal device 10 send terminal identification information, and based on the terminal identification, classify and store the vital data and survey results by type based on the terminal identification information of the terminal device 10 that sent them.

[0056] The learning device 30 generates an emotion estimation model for each type by using vital data classified by type to generate a trained model for each type. After generating the emotion estimation model, learning device 30 stores the generated emotion estimation model in a storage device (step S3). Then, learning device 30 transmits the generated emotion estimation model to information processing device 40. Information processing device 40 stores the emotion estimation model transmitted from learning device 30 in emotion estimation data storage unit 4021.

[0057] Execution Phase In the execution phase, the acquisition unit 404 of the information processing device 40 acquires vital data of each person performing the behavior targeted by the content (step S5). The emotion estimation unit 405 uses the acquired vital data and emotion estimation data (or emotion estimation model) corresponding to the type of behavior included in the content to read emotion data corresponding to the vital data, thereby estimating emotion data of each person (step S6). Alternatively, the emotion estimation unit 405 may input the acquired vital data to an emotion estimation model and obtain emotion data derived by the emotion estimation model, thereby estimating the emotion of each person. The output data generation unit 407 generates output data based on the estimated emotion data (step S7). The output data provision unit 408 transmits the generated output data to an external device.

[0058] Next, the operation of the information processing device 40 will be described. FIG. 8 is a flowchart illustrating the operation of the information processing device 40 in the execution phase. Here, emotion estimation data (emotion estimation model) is generated in advance and stored in emotion estimation data storage unit 4021, and state data is stored in behavior storage unit 4022. When content generation starts, the vital sensor 15 measures the vital data of a person appearing in the content and transmits the measurement results to the terminal device 10. The terminal device 10 transmits the vital data transmitted from the vital sensor 15 to the information processing device 40 together with terminal identification information. The acquisition unit 404 of the information processing device 40 acquires vital data and terminal identification information from the terminal device 10 (step S101). When the vital data is acquired, the emotion estimation unit 405 inputs the acquired vital data into an emotion estimation model corresponding to the behavior included in the content, and obtains emotion data to estimate the emotion (step S102). Here, the emotion estimation unit 405 may select the type of emotion estimation model to be used based on the terminal identification information acquired together with the vital data. Furthermore, the type of emotion estimation model to be used may be determined based on an operation input specifying the type by an administrator from a management device or the like.

[0059] When emotion data is obtained, output data generation unit 407 references behavior memory unit 4022 and generates output data by reading out status data corresponding to the emotion data (step S103). Here, the status data may be data representing performance corresponding to the estimated emotion data. The output data providing unit 408 transmits the output data (e.g., status data) generated by the output data generating unit 407, which includes the emotion data estimated by the emotion estimation unit 405, to the content server 20 and the broadcasting device 50 (step S104).

[0060] When content server 20 receives output data from information processing device 40, it distributes the output data together with the content. Here, content and output data are distributed to terminal device 10 and display device 70 that have requested content distribution from content server 20. Terminal device 10 and display device 70 display the content and output data received from content server 20 on their display screens. When the broadcasting device 50 receives the output data from the information processing device 40, it broadcasts the broadcast content by adding the output data to it. The television receiver 60 receives the content broadcasted from the broadcasting device 50 and displays the received content together with the output data on the display screen.

[0061] Vital data of the characters is acquired, emotional data corresponding to that vital data is estimated, and output data corresponding to the emotional data is displayed along with the content. This allows the results of an analysis of the characters' emotions and psychology to be displayed along with the content, making it possible to provide viewers with new content and new forms of entertainment. In addition, since it is possible to watch sports, games, programs, etc. while viewing the emotions and psychology of the characters (players, etc.) and output data corresponding to these emotions, it is possible to enjoy the content of sports, games, programs, etc. in a more realistic way.

[0062] Such output data may be generated by measuring vital data from a person, analyzing the vital data at regular intervals, and regenerating the output data based on the analysis results. This makes it possible to output output data according to the person's emotions at regular intervals, even if the person's emotions change. Furthermore, vital data may be acquired and analyzed according to the timing at which a person performs a different action, and output data may be regenerated based on the analysis results. In the case of a sport, a different action may be when the defense switches to offense in baseball, when the next batter comes in, or every time the pitcher throws the ball. In the case of a game, it may be every time a move is made in shogi or go. In the case of a television program, it may be every time a scene changes.

[0063] Example 1: Baseball game The above-mentioned content providing system S will be described in the case where video data of a baseball game is distributed as content. FIG. 9 is a conceptual diagram illustrating the flow when a baseball game is distributed as content. When a baseball game starts (step S200), a camera installed in the baseball game venue captures the game situation in the venue (step S201). The area to be captured is, for example, an area including the players participating in the game, and may also be an area including the pitcher.

[0064] Meanwhile, the vital sensors 15 installed at the game venue measure vital data of the participating players (step S202). A plurality of vital sensors 15 may be installed at the game venue, with each vital sensor 15 measuring the vital data of a different player among the plurality of players. Here, for example, one vital sensor 15 measures the vital data of the pitcher. When measuring the vital data of a person participating in a competition, the player may be asked to wear a contact vital sensor, or a non-contact vital sensor may be used. When a non-contact vital sensor is used, the player does not need to wear a vital sensor, and therefore there is no impact on the movements required for competing.

[0065] When vital data of the pitcher is measured, the vital sensor 15 transmits the vital data to the information processing device 40 via the terminal device 10. The acquisition unit 404 of the information processing device 40 acquires the vital data transmitted from the terminal device 10. The emotion estimation unit 405 inputs the acquired vital data into the emotion estimation model to estimate emotion data (step S203) and obtain emotion data (step S204). Here, the emotion estimation model used is an emotion estimation model for the sport "baseball" and the position "pitcher." Then, by estimating emotion data based on the pitcher's vital data, it is possible to estimate emotion data that represents the pitcher's emotions and psychology when pitching.

[0066] The output data generation unit 407 generates output data based on the emotion data (step S205). Here, for example, the emotion data generated is output data for at least one index of "Impatience: XX%", "Calmness: XX%", "Concentration: XX%", "Motivation: XX%", etc. Here, the emotion data index may be expressed as a degree using "%". When the output data is generated, the output data providing unit 408 transmits the output data to the broadcasting device 50.

[0067] When the broadcasting device 50 receives the output data from the information processing device 40, it performs an integration process in which a character string representing the output data is superimposed on an image of the video data generated by the camera, thereby generating integrated content. Here, for example, content is generated in which emotional data is superimposed on a scene of a pitcher pitching. The broadcasting device 50 broadcasts the integrated content.

[0068] Television receiver 60 receives the integrated content broadcast from broadcast device 50 and displays it on the display screen (step S207). This allows viewers to watch the baseball game while visually checking the display screen and understanding the results of an analysis of the pitcher's psychology and emotions, in addition to the commentary based on the commentator's observations.

[0069] Here, we have explained the case where content is broadcast from broadcast device 50, but it is also possible to transmit the integrated content generated by broadcast device 50 to content server 20, and then distribute the integrated content from content server 20 to terminal device 10 and display device 70.

[0070] FIG. 10 is a diagram showing an example of a display screen displayed on the display device 70. As shown in FIG. The display screen of the display device 70 displays content (reference numeral 600) and also displays output data (reference numeral 610). The content is a video (reference numeral 601) captured by a camera showing a pitcher pitching in a baseball game. The output data (reference numeral 610) is displayed and positioned adjacent to the image of the pitcher included in the video (reference numeral 601) so as not to overlap it. Here, the position on the display screen at which the output data is positioned may be a position that overlaps part of the image of the pitcher. The output data may display, for example, "Emotion analysis results of player X" (reference number 611) and "Performance based on current emotion" (reference number 612).

[0071] In this example, the text "Motivation: △△%" is displayed as "Emotional Analysis Results of Player XX." This allows the display of inferred emotions and psychology, which are difficult to grasp from the pitcher's pitching movements, as text, so viewers of the content can not only watch the baseball game, but also understand the inferred emotions and psychology of the player (in this case, the pitcher). This allows viewers to enjoy the game while also learning about the analysis results of the player's emotions as he pitches.

[0072] The output data also displays strings such as "Hits: XX," "Walks: △×," "Hit by Pitch: △△," and "Strikeout: 〇△" as "Performance Based on Current Emotion." Rather than simply displaying past performance, the display reads from the state data the pitcher's past performance based on his current emotions or state of mind. Therefore, even for players whose performance varies depending on their emotions or state of mind, the display shows performance based on the player's current emotions or state of mind. This allows viewers to enjoy the game while also taking into account past performances under similar psychological conditions. Even if the pitcher is unsuccessful in the game, they can watch how the game will progress based on past situations, such as if he is not losing motivation and is likely to strike out.

[0073] Furthermore, in this use example 1, we have described a case where vital data is acquired from a pitcher, the pitcher's emotions are estimated, and output data is generated according to the estimated emotional data. However, when a pitcher and a batter are playing against each other, vital data may be acquired not only from the pitcher but also from the batter, the batter's emotions may be estimated, output data according to the estimated emotional data may be generated, and output together with the content. That is, when a battle action is taking place between a first person and a second person competing against the first person, the emotion estimation unit 405 may estimate emotion data of the first person based on vital data acquired for the first person, and may also estimate emotion data of the second person based on vital data acquired for the second person. Even if the action is other than a baseball game, if it is a competitive sport or game, such as table tennis, shogi, or go, the emotion estimation unit 405 may estimate the emotion of each player from the vital data, and the output data generation unit may generate output data. This allows the emotions and psychology of each competing athlete to be analyzed based on vital data, allowing viewers to watch the content while understanding the emotions and psychology of each competing athlete.

[0074] Furthermore, the behavior memory unit 4022 may store past match results for each combination of emotion data of a first person and emotion data of a second person among the competing characters. The output data generation unit 407 may then read out from the behavior memory unit 4022 the match results corresponding to the combination of emotion data obtained from the first person and emotion data obtained from the second person among the competing characters, and generate output data based on the read match results. This makes it possible to obtain output data indicating what match results were achieved for a combination of emotion data estimated for the first athlete and emotion data estimated for the second athlete. This allows viewers to watch content while understanding what past match results were achieved for each emotion combination of the athletes. For example, if the motivation of both the pitcher and the batter is above a certain level, output data can be obtained that indicates that the batter is more likely to hit a home run, and even if the pitcher is less motivated than the batter, output data can be obtained that indicates that the pitcher is more likely to strike out the batter.Then, for the multiple athletes obtained in this way, past match results according to the combination of estimated emotion data can be displayed together with the content, so viewers can watch the match progress while viewing this output data.

[0075] (Example 2: Soccer match) The above-mentioned content providing system S will be described in the case where video data of a soccer match is distributed as content. In the above example, a case where output data is generated according to emotion data obtained from one person is described, but in this example, a case where output data is generated according to emotion data from multiple people is described. When a soccer match starts, vital data of each player belonging to one team is measured using a plurality of vital sensors 15. Here, non-contact vital sensors 15 are used. The acquisition unit 404 of the information processing device 40 acquires vital data measured by each vital sensor 15 via the terminal device 10. The emotion estimation unit 405 estimates emotion data for each vital data. The emotion estimation unit 405 then estimates the overall emotion or psychology of the team as a whole using the emotion data obtained for each player. For example, the emotion estimation unit 405 may estimate the emotion or psychology of the entire team by calculating the correlation between each player belonging to a team. Alternatively, the emotion estimation unit 405 may calculate the overall degree of the entire team by calculating a cumulative value or an average value of the degree based on emotion data for one indicator of each player belonging to a team, and use this as the estimated emotion data for the entire team.

[0076] The output data generation unit 407 generates output data based on the emotion data of the entire team estimated by the emotion estimation unit 405. Here, the output data generation unit 407 may generate output data according to a combination of the emotion data of each player, or may generate output data according to the emotion or psychology of the entire team estimated by the emotion estimation unit 405 from the emotion data of each player. For example, the emotion estimation unit 405 generates output data such as "the entire team is united in its will to attack," "the entire team is united in its will to defend," "several members of the team are trying to continue attacking without giving up on the game, but the remaining members have given up on the game," "the entire team is tired and unable to concentrate," "the entire team's concentration level for the game: XX%," "the team's motivation to win: XX%," etc.

[0077] Here, the emotional data of the entire team may be associated with evaluation results such as "the entire team is united in its will to attack" or "the entire team is united in its will to defend" and stored in advance in the behavior memory unit 4022 as status data, so that the output data generation unit 407 can read out output data corresponding to the emotional data of the entire team. Here, this evaluation result may be obtained by having the manager evaluate the emotions of each player in a past game and the overall team atmosphere in relation to those emotions, and the evaluation result may be associated with the emotional data and stored in the behavior memory unit 4022. This makes it possible to use output data that takes into account the evaluation of the manager, who has a more detailed understanding of the team's situation, rather than a third party such as a game commentator or play-by-play rep. Furthermore, the output data generation unit 407 may calculate the overall level of the entire team by calculating the cumulative value or average value of the levels based on the emotion data for one index of each player belonging to one team, and use the estimated emotion data of the entire team as output data. In this way, for example, the output data generation unit 407 can obtain indexes and their levels such as "level of concentration of the entire team for the game: XX%" and "team motivation to win: XX%" as output data.

[0078] In this example, the case of soccer has been described, but other sports or games involving cooperation between multiple players may also be used. Examples of other sports include basketball and American football. Examples of other games include team competitions in competitive karuta and team competitions in e-sports.

[0079] <Example 3: Variety shows> The above-mentioned content providing system S will be described in the case where a program is distributed as content. The program may be a program broadcast on television or a program distributed over the Internet. Here, the case where the program is a variety show will be described as an example. The camera captures the program performers. Vital sensor 15 measures the vital data of the program performers. Here, vital sensor 15 may be a contact type and attached to the program performers, or a non-contact type and installed in the program recording studio.

[0080] In a scene in which a TV performer (such as a comedian) is eating piping hot oden and exclaiming, "It's hot! It's hot!", the acquisition unit 404 acquires vital data of the TV performer using a vital sensor. The emotion estimation unit 405 then estimates the emotion data of the TV performer, and the output data generation unit 407 generates output data according to the estimated emotion data. In this case, if data indicating a happiness index of "good" is obtained as the emotion data, the output data generation unit 407 generates a character string such as "Estimated psychology: He seems happy." As a result, in a scene where a program participant says "Hot! Hot!", the text "Estimated psychology: Seems happy" is displayed near the image of the program participant. This allows viewers to understand that although the program participant is saying "Hot!", they are actually feeling happy, and they can enjoy the gap between what is said and what is felt. This makes it possible for variety shows to offer a new form of entertainment.

[0081] In the above-described embodiment, the content may be a live broadcast or live distribution, or may be a broadcast or distribution of recorded content. If the content is recorded, output data can be generated by acquiring vital data from a person using vital sensor 15 during recording, and the output data can be broadcast or distributed together with the content when the content is broadcast or distributed.

[0082] According to the embodiment described above, by making it possible to display output data together with content, the output data is displayed together when the content is viewed, so that new content can be provided that allows the output data to be viewed together with the content. Note that the content and the output data may be provided separately, or content that includes the output data may be broadcast or distributed.

[0083] Although the information processing device 40 has been described as being configured as a device separate from the content server 20, the learning device 30, and the broadcasting device 50, it may also be mounted on any of the content server 20, the learning device 30, and the broadcasting device 50. Also, a program that realizes the functions of the information processing device 40 may be downloaded from the information processing device 40 to the television set 60 or the display device 70, and the television set 60 or the display device 70 may execute this program to display the output data together with the content.

[0084] The information processing device 40 in the above-described embodiment may be implemented by a computer. In this case, a program for implementing the functions may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. The term "computer-readable recording medium" may also include media that dynamically store programs for a short period of time, such as communication lines used when transmitting programs over a network such as the Internet or a telephone line, or media that store programs for a fixed period of time, such as volatile memory within a computer system serving as a server or client. The program may be a program that implements only a portion of the functions described above, or may be a program that can implement the functions described above in combination with a program already stored in the computer system, or may be implemented using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0085] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Explanation of symbols]

[0086] 10, 10a, 10b, 10n terminal equipment 15, 15a, 15b, 15n Vital Sensors 20 Content Server 30 Learning Device 40 Information processing equipment 50 Broadcasting Equipment 60 Television receiver 70 Display device 401,701 Communications Department 402,702 Storage section 403,703 Control section 404 Acquisition Department 405 Emotion estimation part 407 Output Data Generation Unit 408 Output Data Providing Department 704 Input section 705 Output Section 4021 Emotion estimation data storage unit 4022 Behavioral Memory Department NW Network S Contents provision system U1 User U2 users Ua User

Claims

1. an acquisition unit that acquires vital data, which is a measurement result measured by a vital sensor, from a person who may appear in the displayable content; an emotion estimation unit that estimates emotion data representing an emotion of the person based on the acquired vital data; an output data generation unit that generates output data that can be displayed together with the content and that is based on the estimated emotion data; An information processing device having the above.

2. an action storage unit that stores emotion data and state data that represent results according to actions in association with each other; The output data generation unit reading out state data corresponding to the estimated emotion data, and generating output data based on the read out state data; The information processing device according to claim 1 .

3. the content relates to an activity based on one of a competition and a game; The behavior memory unit The status data is stored as the results of either the competition or the game, The emotion estimation unit Estimating emotion data of the person based on vital data acquired while the person is performing the behavior; The output data generation unit A score, which is state data corresponding to the estimated emotion data, is read out, and output data based on the read out score is generated. The information processing device according to claim 2 .

4. the content relates to competitive or game-based activities involving multiple people; The behavior memory unit As the status data, a match result is stored as status data, The emotion estimation unit When a battle action is being performed between a first person and a second person competing against the first person, emotion data of the first person is estimated based on vital data acquired for the first person, and emotion data of the second person is estimated based on vital data acquired for the second person; The output data generation unit A match result corresponding to a combination of the emotion data obtained from the first person and the emotion data obtained from the second person is read from the behavior storage unit, and output data is generated based on the read match result. The information processing device according to claim 3 .

5. the content is a competition or game in which a team can participate, The emotion estimation unit Estimating emotion data of each of the target individuals based on vital data acquired from at least two individuals among a plurality of individuals belonging to the same team; The output data generation unit Based on the emotion data of each of the target people, the psychological state of the team to which the person belongs is estimated, and output data based on the estimated psychological state is generated. The information processing device according to claim 1 .

6. A learning device that learns the relationship between emotions of people who may appear in displayable content and output data that may be displayed together with the content.

7. An information processing device according to any one of claims 1 to 5; A content output device A content providing system including: The information processing device includes: transmitting output data based on emotion data estimated based on vital data acquired when the person appears in the content to a content output device; the content output device, a receiving unit that receives the output data transmitted from the information processing device; a display unit that displays the content from a content distribution device that distributes the content and displays output data transmitted from the information processing device together with the content; A content providing system having the above.

8. 1. A computer-implemented information processing method, comprising: Acquire vital data, which is a measurement result measured by a vital sensor, from a person who may appear in the displayable content; Estimating emotion data representing an emotion of the person based on the acquired vital data; generating output data that can be displayed together with the content and that is based on the estimated emotion data; An information processing method including:

9. A content providing method executed in a content providing system including an information processing device and a content output device, The information processing device includes: Acquire vital data, which is a measurement result measured by a vital sensor, from a person who may appear in the displayable content; Estimating emotion data representing an emotion of the person based on the acquired vital data; generating output data that can be displayed together with the content, the output data being based on the estimated emotion data; When output data based on emotion data estimated based on vital data acquired while the person appears in the content is generated, the generated output data is transmitted to a content output device; the content output device, receiving the output data transmitted from the information processing device; receiving and displaying content from a content distribution device that distributes the content; The output data transmitted from the information processing device is displayed together with the content. A content provision method including:

Images