Systems, methods, server devices, and programs

Abstract

To provide a system capable of protecting personal information of a user, etc., and analyzing the feelings of the user.SOLUTION: A system includes a plurality of terminals and a server device. Each of the plurality of terminals comprises vital data acquisition means, face image acquisition means, and generation means. The vital data acquisition means acquires vital data of a holder from a wearable terminal worn by the holder. The face image acquisition means acquires a face image of the holder. The generation means generates feeling information on the feelings of the holder using the vital data and the face image, and transmits a feeling information notification including the generated feeling information to the server device. The server device comprises reception means and analysis means. The reception means receives a feeling information notification from a terminal. The analysis means analyzes feeling information included in each of the received feeling information notifications and generates analysis results indicative of a change in the entire feelings of the plurality of users.SELECTED DRAWING: Figure 1

Description

[[Technical Field]]

[0001] The present invention relates to a system, a method, a server device and a program. [[Background Art]]

[0002] There is a technique for estimating a user's reaction (emotion) to content presented in a movie theater or the like. For example, Patent Document 1 discloses an information processing apparatus including a reaction information utilization unit that uses reaction information indicating a user's reaction to presented information when the use of the reaction information is permitted. [[Prior Art Documents]] [[Patent Documents]]

[0003] [[Patent Document 1]] International Publication No. 2019 / 073661 [[Summary of the Invention]] [[Problem to be Solved by the Invention]]

[0004] As in Patent Document 1, estimating a user's emotion requires the user's face image or the like. In Patent Document 1, a device managed by a third party different from the user executes processing related to the user's emotion using the face image or the like. In recent years, users' awareness of personal information and privacy has changed. Therefore, many users hesitate (dislike) to provide face images and the like to third parties.

[0005] A main object of the present invention is to provide a system, a method, a server device and a program that contribute to enabling analysis of a user's emotion while protecting the user's personal information and the like. [[Means for Solving the Problem]]

[0006] According to a first aspect of the present invention, a system is provided comprising a plurality of terminals, each owned by a plurality of users, to which the same content is provided, and a server device, wherein each of the plurality of terminals includes vital data acquisition means for acquiring vital data of the owner from a wearable terminal worn by the owner, face image acquisition means for acquiring a face image of the owner by photographing the owner, and generation means for generating emotion information relating to the owner's emotions using the acquired vital data and the acquired face image, and transmitting an emotion information notification including the generated emotion information to the server device, and the server device includes receiving means for receiving the emotion information notifications from each of the plurality of terminals, and analysis means for analyzing the emotion information contained in each of the received emotion information notifications and generating an analysis result showing the change in emotions of the plurality of users as a whole.

[0007] According to a second aspect of the present invention, a system is provided comprising: multiple terminals, each owned by multiple users, to which the same content is provided; a camera device installed to capture the multiple users; and a server device that stores the facial information of each of the multiple users, wherein the server device acquires image data from the camera device, identifies the user corresponding to each of the multiple facial images in the image data by one-to-N matching (N is a positive integer) using the multiple facial images extracted from the image data and the stored facial information, transmits the facial image of the identified user to the terminal of the identified user, each of the multiple terminals acquires vital data of the owner from a wearable device worn by the owner, generates emotion information relating to the owner's emotions using the acquired vital data and the facial image acquired from the server device, transmits an emotion information notification containing the generated emotion information to the server device, the server device receives the emotion information notifications from each of the multiple terminals, analyzes the emotion information contained in each of the received emotion information notifications, and generates an analysis result showing the change in emotions of all of the multiple users.

[0008] According to a third aspect of the present invention, a system is provided comprising: multiple terminals, each owned by a plurality of users, to which the same content is provided; a camera device installed to capture the plurality of users; and a server device, wherein the server device acquires image data from the camera device and transmits the acquired image data to each of the plurality of terminals; each of the plurality of terminals stores the face image of the owner; each of the plurality of terminals identifies the face image of the owner from among the plurality of face images captured in the image data by one-to-N matching (N is a positive integer) using the plurality of face images extracted from the image data and the stored face information of the owner; the owner acquires vital data from the wearable terminal worn by the owner; emotion information relating to the owner's emotions is generated using the acquired vital data and the identified face image of the owner; and emotion information notifications including the generated emotion information are transmitted to the server device; the server device receives the emotion information notifications from each of the plurality of terminals, analyzes the emotion information contained in each of the received emotion information notifications, and generates analysis results showing the change in emotions of the plurality of users as a whole.

[0009] A fourth aspect of the present invention provides a system including a server device and a plurality of terminals, each owned by a plurality of users, to which the same content is provided, wherein each of the plurality of terminals acquires vital data of the owner from a wearable terminal worn by the owner, acquires a facial image of the owner by photographing the owner, generates emotion information relating to the owner's emotions using the acquired vital data and the acquired facial image, transmits an emotion information notification containing the generated emotion information to the server device, the server device receives the emotion information notifications from each of the plurality of terminals, analyzes the emotion information contained in each of the received emotion information notifications, and generates an analysis result showing the change in emotions of the plurality of users as a whole.

[0010] A fifth aspect of the present invention provides a server device comprising: a receiving means for receiving emotion information notifications from multiple terminals, each owned by multiple users, to which the same content is provided. These notifications include emotion information relating to the emotions of each user, generated using facial images obtained by each terminal photographing the user and vital data obtained from wearable terminals worn by each user; and an analysis means for analyzing the emotion information contained in each of the received emotion information notifications and generating an analysis result showing the change in emotions of all the users.

[0011] A sixth aspect of the present invention is provided, which causes a computer mounted on a terminal to perform the following processes: acquiring vital data of the owner from a wearable terminal worn by the owner of the device; acquiring a facial image of the owner by photographing the owner; generating emotion information relating to the owner's emotions using the acquired vital data and the acquired facial image; and transmitting an emotion information notification including the generated emotion information to a server device. [Effects of the Invention]

[0012] From each perspective of the present invention, a system, method, server device, and program are provided that contribute to the analysis of users' emotions while protecting users' personal information. However, the effects of the present invention are not limited to those described above. The present invention may produce other effects in lieu of or in conjunction with the effects described above. [Brief explanation of the drawing]

[0013] [Figure 1] Figure 1 is a diagram illustrating the outline of one embodiment. [Figure 2] Figure 2 is a flowchart illustrating the operation overview of one embodiment. [Figure 3] Figure 3 is a diagram illustrating the schematic of the first embodiment. [Figure 4]FIG. 4 is a diagram showing an example of a schematic configuration of the emotion analysis system according to the first embodiment. [Figure 5] FIG. 5 is a diagram for explaining the operation of the emotion analysis system according to the first embodiment. [Figure 6] FIG. 6 is a diagram for explaining the operation of the emotion analysis system according to the first embodiment. [Figure 7] FIG. 7 is a diagram showing an example of a processing configuration of a terminal according to the first embodiment. [Figure 8] FIG. 8 is a diagram showing an example of display on the terminal according to the first embodiment. [Figure 9] FIG. 9 is a diagram showing an example of display on the terminal according to the first embodiment. [Figure 10] FIG. 10 is a flowchart showing an example of the operation of the terminal according to the first embodiment. [Figure 11] FIG. 11 is a diagram showing an example of a processing configuration of a server device according to the first embodiment. [Figure 12] FIG. 12 is a diagram showing an example of a participant management database according to the first embodiment. [Figure 13] FIGS. 13A and 13B are diagrams showing an example of emotion information according to the first embodiment. [Figure 14] FIG. 14 is a diagram showing an example of an emotion analysis result according to the first embodiment. [Figure 15] FIG. 15 is a sequence diagram showing an example of the operation of the emotion analysis system according to the first embodiment. [Figure 16] FIG. 16 is a diagram for explaining the outline of the second embodiment. [Figure 17] FIG. 17 is a diagram showing an example of a processing configuration of a server device according to the second embodiment. [Figure 18] FIG. 18 is a diagram showing an example of a participant management database according to the second embodiment. [Figure 19] FIG. 19 is a sequence diagram showing an example of the operation of the emotion analysis system according to the second embodiment. [Figure 20]Figure 20 shows an example of the processing configuration of a server device according to a modified example of the second embodiment. [Figure 21] Figure 21 is a diagram showing an example of the hardware configuration of the terminal disclosed in this application. [Modes for carrying out the invention]

[0014] First, an overview of one embodiment will be described. The reference numerals in the drawings attached to this overview are provided for convenience as examples to aid understanding, and this overview is not intended to be limiting in any way. Furthermore, unless otherwise specified, the blocks shown in each drawing represent functional units, not hardware units. The connecting lines between blocks in each drawing include both bidirectional and unidirectional lines. Unidirectional arrows schematically indicate the flow of the main signal (data) and do not exclude bidirectional flow. In this specification and in the drawings, elements that can be similarly described are denoted by the same reference numerals, thus omitting redundant explanations.

[0015] A system according to one embodiment includes a plurality of terminals 101 and a server device 102 (see Figure 1). The plurality of terminals 101 are owned by multiple users, each of whom is provided with the same content. Each of the plurality of terminals 101 is equipped with vital data acquisition means 111, face image acquisition means 112, and generation means 113. The vital data acquisition means 111 acquires the owner's vital data from a wearable terminal worn by the owner (step S1 in Figure 2). The face image acquisition means 112 acquires the owner's face image by photographing the owner (step S2). The generation means 113 generates emotion information regarding the owner's emotions using the acquired vital data and the acquired face image, and transmits an emotion information notification including the generated emotion information to the server device 102 (step S3). The server device 102 is equipped with receiving means 121 and analysis means 122. The receiving means 121 receives emotion information notifications from each of the plurality of terminals 101 (step S4). The analysis means 122 analyzes the emotional information contained in each received emotional information notification and generates an analysis result showing the change in emotions of the entire group of users (step S5).

[0016] Each of the multiple terminals 101 acquires a facial image of the owner and calculates emotional information (emotion score) using the acquired facial image and vital data obtained from the wearable device. In other words, the process of acquiring the owner's facial image and the process of calculating emotional information using the owner's personal information (facial image, vital data) are completed within the terminal 101. Therefore, the above personal information is not passed to the external server device 102. Accordingly, the user's personal information is properly protected. The server device 10 also performs analysis processing using the emotional information of each terminal owner. As a result, a system is provided that allows for the analysis of the user's emotions while protecting the user's personal information.

[0017] Specific embodiments will be described in more detail below with reference to the drawings.

[0018] [First Embodiment] The first embodiment will be described in more detail with reference to the drawings.

[0019] In the emotion analysis system (information processing system) according to the first embodiment, the same content is provided to multiple users, such as in a movie or seminar. In the first embodiment, for example, as shown in Figure 3, a seminar by a speaker is assumed to be conducted. The emotion analysis system analyzes the emotions (impressions) of all the seminar participants.

[0020] Specifically, the sentiment analysis system calculates points (times) when audience members show interest in the provided content, as well as points where they become bored. The sentiment analysis system then provides the calculated results to seminar organizers, speakers, and other relevant parties.

[0021] [System Configuration] Figure 4 shows an example of a schematic configuration of an emotion analysis system according to the first embodiment. As shown in Figure 4, the emotion analysis system includes a server device 10.

[0022] For example, server device 10 is managed and operated by a company or other organization hosting a seminar. Server device 10 may be installed within the company's building or on a network (on the cloud).

[0023] Each seminar participant wears a wearable device 20. For example, each participant is given a wearable device 20 when they enter the seminar venue. Alternatively, each participant may already own a wearable device 20.

[0024] Each participant (system user) of a seminar or similar event, where the same content is provided, possesses a terminal 30. Terminal 30 is, for example, a mobile device such as a smartphone or tablet.

[0025] The wearable terminal 20 and terminal 30 are configured to communicate with each other using a short-range wireless communication method such as Bluetooth®. Furthermore, terminal 30 is configured to communicate with the server device 10 via a wireless LAN (Local Area Network) or mobile network.

[0026] The configuration of the emotion analysis system shown in Figure 4 is illustrative and not intended to limit its configuration. For example, the system may include multiple server devices 10. Also, although Figure 4 illustrates three audience members, it is not intended to limit the number of users of the emotion analysis system to three.

[0027] [General operation] Next, we will describe the general operation of the emotion analysis system according to the first embodiment.

[0028] <Participant Registration> Seminar participants register before the seminar begins. Specifically, seminar participants access the server device 10 by operating terminal 30. Participants register on the web page provided by the server device 10.

[0029] For example, seminar participants operate terminal 30 to input their user ID into server device 10 (see Figure 5). For example, an email address that terminal 30 can receive emails from is entered into server device 10 as the user ID.

[0030] Alternatively, seminar participants may input information such as their gender and affiliated company into the server device 10. The server device 10 manages the information entered by seminar participants along with their user IDs.

[0031] <Acquisition of vital data> The wearable device 20 acquires vital data from the wearer (seminar participant, etc.). For example, the wearable device 20 acquires vital signs such as pulse, blood pressure, respiration, and body temperature while the seminar is being held.

[0032] The wearable device 20 transmits the acquired vital data to the terminal 30 periodically or at predetermined intervals. The wearable device 20 transmits the acquired vital data to the terminal 30 held by the user wearing the device (see Figure 6).

[0033] <Acquiring facial images> Each user's terminal 30 is set up to photograph its owner throughout the seminar. Terminal 30 is set up to photograph the owner's face. Terminal 30 continues to photograph the owner's face until the seminar ends.

[0034] Terminal 30 acquires image data including the face image of the owner (seminar participant) by taking a picture of the owner's face. Terminal 30 extracts face images from the acquired image data. At that time, terminal 30 extracts face images that are determined to be the same person's face image as face images pre-stored in its own device.

[0035] <Calculation of emotional information> Terminal 30 calculates information about the emotions of seminar participants (emotional information) using vital data acquired from wearable terminal 20 and facial images obtained through photography.

[0036] Specifically, terminal 30 uses vital data and facial images to calculate an emotion score that quantifies the feelings of seminar participants, such as "enjoyable" or "boring." For example, terminal 30 calculates the emotion score in real time using the vital data and facial images of seminar participants. For example, terminal 30 calculates time-series data of emotion scores that quantify the user's feelings, such as time t1 = +10 (extremely enjoyable) and time t2 = -10 (extremely boring).

[0037] When the seminar ends, terminal 30 sends an emotion information notification to server device 10, which includes the calculated emotion information (time-series data of emotion scores) and the user ID. Before sending the emotion information to server device 10, terminal 30 obtains consent (consent from the owner) regarding the transmission of the calculated emotion information. This consent may also be obtained at the start of the seminar.

[0038] <Emotion analysis> The server device 10 analyzes the emotions of seminar participants using the acquired emotional information. The server device 10 generates analysis results that include changes in the emotions of all seminar participants (average participants). Specifically, the server device 10 uses the acquired emotional information to calculate information such as the time periods when many audience members were interested and the time periods when many audience members felt bored, as "emotional analysis results."

[0039] The calculated sentiment analysis results will be provided to the seminar organizers, etc. The seminar organizers, etc., will use the sentiment analysis results to improve their operations (improvement of seminar content).

[0040] Next, we will describe the details of each device included in the emotion analysis system according to the first embodiment.

[0041] [Terminal] Examples of terminal 30 include mobile devices such as smartphones, cell phones, game consoles, and tablets. Terminal 30 can be any device as long as it can receive user input and communicate with the wearable terminal 20 and the server device 10.

[0042] Figure 7 shows an example of the processing configuration (processing module) of terminal 30 according to the first embodiment. Referring to Figure 7, terminal 30 comprises a communication control unit 201, an original face image management unit 202, a participant registration control unit 203, a vital data acquisition unit 204, a face image acquisition unit 205, an emotion information generation unit 206, and a storage unit 207.

[0043] The communication control unit 201 is a means for controlling communication with other devices. For example, the communication control unit 201 receives data (packets) from the server device 10. The communication control unit 201 also transmits data to the server device 10. The communication control unit 201 passes the data received from other devices to other processing modules. The communication control unit 201 transmits the data obtained from other processing modules to other devices. In this way, other processing modules send and receive data with other devices via the communication control unit 201. The communication control unit 201 has the function of a receiving unit that receives data from other devices and the function of a transmitting unit that transmits data to other devices.

[0044] The communication control unit 201 supports short-range wireless communication such as Bluetooth® and communicates with the wearable terminal 20 using this short-range wireless communication. The pairing process required when terminal 30 and wearable terminal 20 communicate via Bluetooth® is performed, for example, when a seminar participant receives the wearable terminal 20.

[0045] The original facial image management unit 202 is a means of managing the facial image of the owner of the terminal 30. The original facial image management unit 202 acquires the original facial image (hereinafter referred to as the original facial image) which will be used as the original facial information for various authentication and matching processes. For example, the original facial image management unit 202 acquires the original facial image using a GUI (Graphical User Interface) as shown in Figure 8.

[0046] The original facial image management unit 202 stores the acquired original facial image in the storage unit 207. At that time, the original facial image management unit 202 may encrypt the original facial image and store the encrypted original facial image in the storage unit 207.

[0047] Furthermore, the original face image management unit 202 may generate feature quantities from the acquired original face image and store the generated feature quantities in the storage unit 207. In the following description, the feature quantities generated from the original face image will be referred to as "original feature quantities". The storage unit 207 stores the face information of the terminal owner (original face image, original feature quantities).

[0048] Regarding the feature generation process, existing technologies can be used, so a detailed explanation will be omitted. For example, the original face image management unit 202 extracts the eyes, nose, mouth, etc., from the face image as feature points. Then, the original face image management unit 202 calculates the position of each feature point and the distance between each feature point as feature quantities (generating a feature vector consisting of multiple feature quantities).

[0049] The participant registration control unit 203 is a means for performing control related to user participant registration. The participant registration control unit 203 accesses the web page provided by the server device 10 in response to the user's (seminar participant's) actions.

[0050] The participant registration control unit 203 obtains information entered by the user into the GUI provided by the server device 10. The participant registration control unit 203 then sends the obtained information (for example, an email address) to the server device 10.

[0051] The vital data acquisition unit 204 is a means for controlling the acquisition of vital data. The vital data acquisition unit 204 acquires vital data of the wearer from the wearable terminal 20 worn by the wearer of terminal 30.

[0052] Specifically, the vital data acquisition unit 204 receives vital data from a wearable terminal 20 that is in a state where it can communicate with its own device (terminal 30). More specifically, the vital data acquisition unit 204 acquires vital data from a wearable terminal 20 that has completed pairing.

[0053] The vital data acquisition unit 204 stores the acquired vital data in the storage unit 207. The vital data acquisition unit 204 stores the acquired vital data (specific numerical values ​​such as pulse rate) in association with the time of vital data acquisition in the storage unit 207.

[0054] The face image acquisition unit 205 is a means for controlling the acquisition of face images. The face image acquisition unit 205 acquires a face image of the owner of the terminal 30 by photographing the owner. The face image acquisition unit 205 starts photographing the user in response to a predetermined action by the user (for example, pressing the seminar start button).

[0055] The facial image acquisition unit 205 controls the camera mounted on the device and continuously photographs users participating in the seminar. The facial image acquisition unit 205 acquires the image data obtained through the photography.

[0056] The face image acquisition unit 205 attempts to extract face images (face regions) from the acquired image data.

[0057] Note that the face image acquisition unit 205 can use existing technologies for extracting face images, so a detailed explanation is omitted. For example, the face image acquisition unit 205 may use a trained model trained by a CNN (Convolutional Neural Network) to extract face images (face regions) from image data. Alternatively, the face image acquisition unit 205 may use methods such as template matching to extract face images.

[0058] If the facial image extraction is successful, the facial image acquisition unit 205 performs a one-to-one match using the facial information stored in the storage unit 207 and the facial image obtained through capture. Specifically, the facial image acquisition unit 205 generates feature quantities from the extracted facial image. The facial image acquisition unit 205 then performs a one-to-one match using the generated feature quantities and the original feature quantities (feature quantities generated from the original facial image).

[0059] At that time, the face image acquisition unit 205 calculates the similarity between the two feature quantities. The chi-squared distance, Euclidean distance, etc., can be used to calculate this similarity. The greater the distance, the lower the similarity; the closer the distance, the higher the similarity.

[0060] If the similarity is greater than a predetermined value (if the distance is shorter than a predetermined value), the face image acquisition unit 205 determines that the one-to-one matching (one-to-one authentication) was successful. If the similarity is less than or equal to the predetermined value, the face image acquisition unit 205 determines that the one-to-one matching failed.

[0061] If one-to-one matching is successful, the face image acquisition unit 205 stores the face image extracted from the image data in the storage unit 207. The face image acquisition unit 205 stores the acquired face image and the time of face image acquisition in the storage unit 207 in association. Face images obtained through photography that have successfully undergone one-to-one matching are used to generate emotion information.

[0062] If one-to-one matching fails, the face image acquisition unit 205 does not perform any special processing. The face image acquisition unit 205 discards the face image extracted from the image data.

[0063] The emotion information generation unit 206 is a means for generating emotion information. The emotion information generation unit 206 generates emotion information about the emotions of the owner of the terminal 30 using the acquired vital data and the acquired facial image. The emotion information generation unit 206 transmits an emotion information notification containing the generated emotion information to the server device 10.

[0064] For example, the emotion information generation unit 206 calculates emotion information (emotion score) in real time while the seminar is being held. For example, the emotion information generation unit 206 calculates the emotion score when vital data or facial images are acquired.

[0065] The emotion information generation unit 206 calculates the emotion score of the user (owner of terminal 30; seminar participant) using vital data and facial images. For example, the emotion information generation unit 206 obtains the emotion score using a learning model obtained through machine learning.

[0066] For example, the emotion information generation unit 206 obtains an emotion score using a learning model that outputs an emotion score when vital data and a facial image are input during or immediately before the emotion score calculation.

[0067] The learning model is obtained through machine learning using training data in which a large amount of data (a combination of vital data and facial images) is labeled (emotion score; for example, a numerical value on a 21-point scale ranging from +10 to -10). Any algorithm such as support vector machines, boosting, or neural networks can be used to generate the learning model. Note that the above-mentioned algorithms such as support vector machines can be used using publicly known techniques, so their explanation will be omitted.

[0068] For example, a positive emotion score indicates positive emotions (joy, happiness, interest, excitement, etc.), while a negative emotion score indicates negative emotions (anger, sadness, boredom, indifference, etc.). Furthermore, the value of the emotion score indicates the degree of the emotion.

[0069] The emotion information generation unit 206 may calculate an emotion score using a learning model for vital data and a learning model for face images, respectively. For example, the emotion information generation unit 206 obtains an emotion score (vital) by inputting vital data into the learning model for vital data. Alternatively, the emotion information generation unit 206 obtains an emotion score (face image) by inputting a face image into the learning model for face images.

[0070] The emotion information generation unit 206 may calculate the overall emotion score by performing arithmetic operations on the two obtained emotion scores (vital signs and facial image). For example, the emotion information generation unit 206 calculates the overall emotion score by adding the two emotion scores. Alternatively, the emotion information generation unit 206 may calculate the overall emotion score by assigning weights to the two emotion scores and adding them together. Alternatively, the emotion information generation unit 206 obtains the overall emotion score by multiplying the two emotion scores.

[0071] In this way, the emotion information generation unit 206 inputs vital data and facial images into a learning model obtained through machine learning to acquire an emotion score that quantifies the emotions of the owner of the terminal 30.

[0072] The emotion information generation unit 206 may calculate the emotion score using table information containing emotion scores. For example, the emotion information generation unit 206 may calculate the emotion score (vital) using table information that stores vital data (pulse rate, blood pressure, etc.) in association with emotion scores.

[0073] The emotion information generation unit 206 stores the calculated emotion score in the storage unit 207, associating it with the time the score was calculated. The emotion information generation unit 206 stores the time-series data of the emotion score as emotion information in the storage unit 207.

[0074] The emotion information generation unit 206 transmits the generated emotion information (time-series data of emotion scores) to the server device 10. For example, the emotion information generation unit 206 transmits the emotion information to the server device 10 at the end of the seminar or when a user leaves the seminar venue.

[0075] For example, when the emotion information generation unit 206 detects a predetermined action by the user (e.g., pressing the seminar end button), it transmits emotion information to the server device 10. At that time, the emotion information generation unit 206 obtains the user's consent to provide the emotion information to a third party (e.g., the seminar organizing company).

[0076] For example, the emotion information generation unit 206 obtains the user's intention (consent or refusal to provide emotion information) using a GUI as shown in Figure 9.

[0077] If consent to provide emotional information is not obtained, the emotional information generation unit 206 will not perform any special actions.

[0078] Once consent is obtained to provide emotional information, the emotional information generation unit 206 transmits the emotional information to the server device 10. Specifically, the emotional information generation unit 206 transmits an emotional information notification to the server device 10 that includes the calculated emotional information (time-series data of emotional scores) and the user ID (e.g., email address).

[0079] Thus, the emotion information generation unit 206 sends an emotion information notification to the server device 10 when it obtains the consent of the owner of the terminal 30 to send the emotion information to the server device 10.

[0080] The memory unit 207 is a means for storing information necessary for the operation of the terminal 30.

[0081] The operation of terminal 30 described above can be summarized as shown in the flowchart in Figure 10.

[0082] Terminal 30 acquires vital data from the wearable terminal 20 (step S101).

[0083] Terminal 30 acquires the facial image of the user (seminar participant) (step S102).

[0084] Terminal 30 calculates emotional information (emotion score) using vital data and facial images (step S103).

[0085] Before transmitting the emotional information, terminal 30 obtains the user's consent to provide the emotional information to others (step S104).

[0086] Terminal 30 sends an emotion information notification, including the user ID and emotion information, to the server device 10 (step S105).

[0087] [Server equipment] Figure 11 shows an example of the processing configuration (processing module) of the server device 10 according to the first embodiment. Referring to Figure 11, the server device 10 comprises a communication control unit 301, a participant management unit 302, an emotion information processing unit 303, an emotion analysis unit 304, and a storage unit 305.

[0088] The communication control unit 301 is a means for controlling communication with other devices. For example, the communication control unit 301 receives data (packets) from terminal 30. The communication control unit 301 also transmits data to terminal 30. The communication control unit 301 passes data received from other devices to other processing modules. The communication control unit 301 transmits data acquired from other processing modules to other devices. In this way, other processing modules send and receive data with other devices via the communication control unit 301. The communication control unit 301 has the function of a receiving unit that receives data from other devices and the function of a transmitting unit that transmits data to other devices.

[0089] The participant management unit 302 is a means of managing information regarding participants in seminars and other events.

[0090] When a participant performs a specified operation on a web page or the like, the participant management unit 302 obtains information about that participant (for example, an email address that will serve as the user ID). The participant management unit 302 obtains the above information about the participant using a GUI or the like.

[0091] The participant management unit 302 stores the acquired information (including user IDs) in the participant management database (see Figure 12). Note that the participant management database shown in Figure 12 is an example and is not intended to limit the items to be stored. For example, the date and time of participant registration may also be stored in the participant management database.

[0092] The emotion information processing unit 303 is a means for managing and processing the emotion information of users (seminar participants). The emotion information processing unit 303 processes emotion information notifications received from terminal 30.

[0093] Upon receiving an emotion information notification, the emotion information processing unit 303 stores the emotion information in the storage unit 305 for each user ID included in the emotion information notification. The emotion information processing unit 303 also stores time-series data of the emotion score for each participant.

[0094] For example, the memory unit 305 stores time-series data of emotional scores, as shown in Figures 13A and 13B. Figure 13A is a graph of the time-series data of participant A's emotional score. Figure 13B is a graph of the time-series data of participant B's emotional score.

[0095] The emotion analysis unit 304 is a means for analyzing the emotions of seminar participants. The emotion analysis unit 304 analyzes the emotion information contained in each emotion information notification received from multiple terminals 30 and generates an analysis result that shows the changes in the emotions of all users participating in the seminar.

[0096] In other words, the emotion analysis unit 304 analyzes the emotional information of each participant (time-series data of emotional scores) to generate an analysis result of emotions common to all seminar participants (many seminar participants) at each point and time during the seminar.

[0097] Specifically, the emotion analysis unit 304 analyzes the emotions (changes in emotions) of all participants in response to requests from system administrators, etc. The emotion analysis unit 304 then presents the analysis results to the system administrators, etc.

[0098] For example, the emotion analysis unit 304 analyzes the emotions of all participants using emotional information (time-series data of emotional scores) obtained from each participant.

[0099] For example, the emotion analysis unit 304 calculates the average emotional score of each participant at predetermined points from the start to the end of the seminar. For example, the emotion analysis unit 304 calculates the average emotional score of each participant at one-minute intervals from the start of the seminar.

[0100] If there are participants for whom an emotion score has not been obtained at the predetermined time, the emotion analysis unit 304 will use the most recent emotion score or use an emotion score interpolated using the preceding and succeeding emotion scores.

[0101] Alternatively, the emotion analysis unit 304 calculates the average emotion score for each participant over a predetermined period (e.g., 5 minutes). The emotion analysis unit 304 may use the calculated average emotion scores to calculate the average emotion score for all seminar participants.

[0102] For example, the emotion analysis unit 304 calculates the average emotion score for participant A during the first 5 minutes of the seminar. Similarly, the emotion analysis unit 304 calculates the average emotion score for participant B during the first 5 minutes of the seminar. The emotion analysis unit 304 then calculates the average of the average emotion scores of participant A and participant B to determine the total emotion score for all participants (two participants in the example above) during the first 5 minutes of the seminar.

[0103] In this way, the emotion analysis unit 304 generates analysis results by calculating the average value of the emotion scores of multiple users at a predetermined time or period.

[0104] The emotion analysis unit 304 presents the obtained analysis results (time-series data showing the emotional changes of all participants) to the system administrator or other relevant parties.

[0105] For example, the emotion analysis unit 304 displays the emotion analysis results, as shown in Figure 14, on an LCD monitor or the like. Alternatively, the emotion analysis unit 304 may print the emotion analysis results or transmit them to a terminal held by the system administrator.

[0106] Referring to the results shown in Figure 14, the system administrator (seminar organizer) recognizes that there is room for improvement in the seminar content from time t3 to time t6.

[0107] Alternatively, the emotion analysis unit 304 may generate analysis results indicating the time periods when seminar participants showed the most interest in the provided content and the time periods when they felt the most bored. For example, in the example in Figure 14, time t1-t2 is generated as the time period when many participants showed the most interest in the provided content, and time t4-t5 is generated as the time period when many participants felt the most bored.

[0108] The memory unit 305 is a means for storing information necessary for the operation of the server device 10.

[0109] [System operation] Next, the operation of the emotion analysis system according to the first embodiment will be described.

[0110] Figure 15 is a sequence diagram showing an example of the operation of the emotion analysis system according to the first embodiment. The operation of the emotion analysis system according to the first embodiment will be explained with reference to Figure 15.

[0111] Terminal 30 acquires vital data from the wearable terminal 20 (step S01).

[0112] Terminal 30 acquires a facial image by taking a picture of the user (step S02).

[0113] Terminal 30 calculates an emotion score using vital data and facial images (step S03).

[0114] Terminal 30 sends an emotion information notification containing the user ID and emotion information to the server device 10 when the seminar ends (step S04).

[0115] The server device 10 stores the emotional information (time-series data of emotional scores) of each participant (step S05).

[0116] The server device 10 performs sentiment analysis on all seminar participants in response to requests from system administrators, etc. (Step S06).

[0117] <Modification example 1 of the first embodiment> In the first embodiment described above, the terminal 30 was described as inputting the user's email address as the user ID to the server device 10. However, the terminal 30 may generate a user ID for the user and send the generated user ID to the server device 10.

[0118] For example, the participant registration control unit 203 may calculate a hash value unique to the user, such as their name or email address, and send the calculated hash value to the server device 10 as the user ID.

[0119] Alternatively, the seat number of the seat where the seminar participant sits may be used as the user ID. In this case, the seminar participant operates terminal 30 to input the seat number attached to the seat where they are sitting into server device 10. Alternatively, terminal 30 may read the seat number from a 2D barcode attached to the seat and transmit the read seat number to server device 10.

[0120] Alternatively, when participants register, the server device 10 may generate a user ID that uniquely identifies each participant. The server device 10 may then send the generated user ID to the terminal 30 of the user who has indicated their intention to participate in the seminar.

[0121] <Modification example 2 of the first embodiment> The emotion information (time-series data of emotion scores) generated by terminal 30 may be provided to seminar participants. For example, when the emotion information generation unit 206 detects a predetermined action by the user (for example, pressing the seminar feedback display button), it may display the emotion information shown in Figures 13A and 13B.

[0122] <Modification example 3 of the first embodiment> Instead of terminal 30 generating the emotional information, the server device 10 may generate it. In this case, terminal 30 only needs to send vital data acquired from the wearable terminal 20 and facial images obtained through photography to the server device 10 along with the user ID.

[0123] As described above, in the emotion analysis system according to the first embodiment, terminal 30 acquires a facial image of the possessor (owner) and calculates emotion information (emotion score) using the acquired facial image and vital data acquired from the wearable terminal 20. In other words, the process of acquiring the possessor's facial image and the process of calculating the emotion score using the possessor's personal information (facial image, vital data) are completed within terminal 30. Therefore, the above personal information is not transmitted to an external server device 10 or the like. Accordingly, the user's personal information is properly protected.

[0124] Furthermore, terminal 30 performs emotional analysis (calculation of emotional score) of the user using vital data and facial images, allowing for more accurate analysis compared to using either one of them.

[0125] Furthermore, terminal 30 uses pre-registered facial information (original facial image, original features) to determine whether the facial image in the captured image belongs to the terminal owner. As a result, even if someone else is captured in the image because the owner leaves their seat, sentiment analysis (calculation of sentiment score) will not be performed using the facial image of that other person.

[0126] Furthermore, the server device 10 identifies users using their user ID (e.g., email address) and analyzes emotional information (time-series data of emotional scores) obtained from each user's terminal 30. By analyzing the emotional information of each participant, the server device 10 generates information useful for improving the operations of seminar organizers. In addition, the analysis process of the server device 10 is light in processing load, such as calculating the average value of emotional scores. Therefore, even if the number of seminar participants increases significantly, the server device 10 will not be subjected to an excessive load.

[0127] [Second Embodiment] Next, a second embodiment will be described in detail with reference to the drawings.

[0128] In the first embodiment, it was explained that a camera mounted on terminal 30 photographs each seminar participant (the owner of terminal 30). In the second embodiment, it is explained that a camera capable of photographing the entire audience participating in the seminar photographs the seminar participants.

[0129] The following will focus on explaining the differences between the first and second embodiments.

[0130] The emotion analysis system according to the second embodiment includes a camera device 40, as shown in Figure 16. The camera device 40 is installed to acquire an overhead image of the entire seminar venue. The camera device 40 is connected to the server device 10 by wire or wireless connection.

[0131] Figure 17 shows an example of the processing configuration (processing module) of the server device 10 according to the second embodiment. Referring to Figure 17, the participant identification unit 306 is added to the configuration of the server device 10 according to the first embodiment.

[0132] The participant identification unit 306 is a means for performing control related to the identification of seminar participants.

[0133] In the second embodiment, the terminal 30 (participant registration control unit 203) transmits original feature quantities to the server device 10 when a participant is registered. If the user explicitly consents, the terminal 30 may also transmit the user's original facial image to the server device 10. Upon obtaining the original facial image, the server device 10 generates original feature quantities from the original facial image.

[0134] The server device 10 (participant management unit 302) stores the generated original feature quantities in the participant management database (see Figure 18).

[0135] When the seminar begins, the participant identification unit 306 of the server device 10 acquires image data (image data showing all seminar participants) from the camera device 40 periodically or at predetermined intervals.

[0136] The participant identification unit 306 extracts face images from the acquired image data. For example, if there are 10 seminar participants, 10 face images will be extracted.

[0137] The participant identification unit 306 identifies the corresponding seminar participant for each extracted face image. Specifically, the participant identification unit 306 generates feature vectors from the extracted face images. The participant identification unit 306 sets the generated feature vectors on the matching side and the feature vectors stored in the participant management database (original feature vectors) on the registration side, and performs a one-to-many matching (N is a positive integer, the same applies hereafter).

[0138] The participant identification unit 306 calculates the similarity between the feature quantity to be matched and each of the multiple feature quantities on the registered side. If, among the multiple feature quantities registered in the participant management database, there is no feature quantity whose similarity to the feature quantity to be matched is greater than or equal to a predetermined value, the participant identification unit 306 determines that the matching process has failed.

[0139] The participant identification unit 306 determines that the matching process is successful if, among the multiple features registered in the participant management database, there is a feature whose similarity to the feature to be matched is equal to or greater than a predetermined value. The participant identification unit 306 identifies the participant with the entry that has the highest similarity as the participant corresponding to the face image to be matched.

[0140] When a seminar participant corresponding to a facial image is identified through one-to-many matching, the participant identification unit 306 transmits the facial image to the terminal 30 held by the identified participant. Specifically, the participant identification unit 306 transmits a "facial image detection notification" to the terminal 30, which includes the identified participant's user ID, facial image, and the time the facial image was acquired.

[0141] The participant identification unit 306 repeats the participant identification process and face image transmission process described above for each face image included in the image data acquired from the camera device 40.

[0142] The emotion information generation unit 206 of terminal 30 receives a face image detection notification. The emotion information generation unit 206 calculates emotion information (emotion score) using the face image included in the received face image detection notification and vital data acquired from the wearable terminal 20.

[0143] In this manner, the server device 10 acquires image data from the camera device 40. For each of the multiple face images captured in the image data, the server device 10 identifies the user corresponding to each face image by performing a one-to-many matching using the multiple face images extracted from the image data and the stored face information. The server device 10 transmits the identified user's face image to the identified user's terminal 30. The terminal 30 generates emotion information regarding the owner's emotions using vital data and the face image acquired from the server device 10.

[0144] [System operation] Next, the operation of the emotion analysis system according to the second embodiment will be described.

[0145] Figure 19 is a sequence diagram showing an example of the operation of the emotion analysis system according to the second embodiment. The operation of the emotion analysis system according to the second embodiment will be explained with reference to Figure 19. Note that the same processes are assigned the same symbols (steps) in Figures 15 and 19, and their explanations are omitted.

[0146] The server device 10 identifies the seminar participants captured in the image data obtained from the camera device 40 (step S11).

[0147] The server device 10 sends a face image detection notification containing the face image of the identified seminar participant to the terminal 30 of the identified seminar participant (step S12).

[0148] Terminal 30 calculates an emotion score using the acquired vital data and facial image, similar to the first embodiment (step S03).

[0149] Thus, in the second embodiment, the original feature quantities and user ID of the user are transmitted to the server device 10 before the start of the seminar, etc. The server device 10 identifies the seminar participants by one-to-many matching using the original feature quantities and the feature quantities of the face image obtained from the image data captured by the camera device 40. The server device 10 transmits the face image of the identified seminar participant to the terminal 30 (the terminal 30 identified by the user ID).

[0150] <Modification example 1 of the second embodiment> In the second embodiment described above, a case was described in which a one-to-many matching for identifying participants is performed on the server device 10. However, the one-to-many matching may also be performed on the terminal 30 held by each seminar participant.

[0151] In this case, the participant identification unit 306 transmits the image data acquired from the camera device 40 to each seminar participant's terminal 30.

[0152] The face image acquisition unit 205 of terminal 30 extracts face images from image data acquired from the server device 10. For example, if there are 10 seminar participants, 10 face images will be extracted.

[0153] The face image acquisition unit 205 generates feature quantities from each extracted face image. The face image acquisition unit 205 performs a matching process using the original feature quantities stored in the storage unit 207 and the generated feature quantities.

[0154] Specifically, the face image acquisition unit 205 sets the original feature quantities on the matching side and the feature quantities generated from the image data on the registration side, and performs a one-to-many matching. The face image acquisition unit 205 treats the face images (face images extracted from the image data) corresponding to the feature quantities identified by the one-to-many matching as the face images of the owner.

[0155] The face image acquisition unit 205 stores the face image identified by one-to-many matching and the face image acquisition time in the storage unit 207.

[0156] Terminal 30 performs calculation processing of emotional information (emotion score) using vital data and facial images, similar to the first embodiment.

[0157] In this manner, the server device 10 acquires image data from the camera device 40 and transmits the acquired image data to each of the multiple terminals 30. Each of the multiple terminals 30 stores the corresponding face image of the owner. Furthermore, each terminal 30 identifies the owner's face image from among the multiple face images captured in the image data by performing a one-to-many matching (N is a positive integer) using multiple face images extracted from the received image data and the owner's face information stored in the storage unit 207. The terminal 30 then calculates an emotion score using the identified face image.

[0158] <Modification 2 of the second embodiment> In the second embodiment described above, the calculation of emotional information (emotion score) was performed on the terminal 30. However, the calculation of said emotional information may also be performed on the server device 10.

[0159] In this case, the server device 10 includes a vital data processing unit 307 and an emotion information generation unit 308, as shown in Figure 20. Note that the server device 10 does not include the emotion information processing unit 303 described in the first embodiment above.

[0160] The vital data processing unit 307 is a means for processing vital data received from the terminal 30. The emotion information generation unit 308 operates in the same manner as the emotion information generation unit 206 of the terminal 30 described in the first embodiment, etc.

[0161] In the second embodiment, the vital data acquisition unit 204 of the terminal 30 acquires vital data from the wearable terminal 20 and transmits the acquired vital data to the server device 10.

[0162] For example, each time the vital data acquisition unit 204 acquires vital data from the wearable terminal 20, it sends a vital data notification to the server device 10 that includes the seminar participant's user ID, the acquired vital data, and the time the vital data was acquired.

[0163] The vital data processing unit 307 of the server device 10, upon receiving a vital data notification, stores the user ID, vital data, and vital data acquisition time included in the notification.

[0164] The emotion information generation unit 308 calculates emotion information (time-series data of emotion scores) using the facial images of seminar participants identified by the participant identification unit 306 and vital data acquired from the terminal 30.

[0165] Furthermore, the emotion information generation unit 308 uses the user ID to associate (match) the facial image acquired by the participant identification unit 306 with the vital data acquired from the terminal 30.

[0166] <Modification example 3 of the second embodiment> Some functions of the server device 10 according to the second embodiment described above may be performed by the camera device 40.

[0167] For example, the camera device 40 may perform processes such as extracting face images from image data and generating feature quantities from the extracted face images.

[0168] As described above, the emotion analysis system according to the second embodiment includes a camera device 40 capable of acquiring image data that provides an overview of the entire venue. The server device 10 and terminal 30 identify the terminal holder through one-to-many authentication using the facial images captured in the image data obtained from the camera device 40. The terminal holder's terminal 30 uses the identified facial images to calculate an emotion score. As a result, even in seminar venues or other locations without desks where it is difficult to position the terminal 30 to photograph users, the user's emotion score can be calculated and emotional information can be analyzed.

[0169] Next, we will describe the hardware of each device that makes up the emotion analysis system. Figure 21 shows an example of the hardware configuration of terminal 30.

[0170] Terminal 30 can be configured using an information processing device (a so-called computer), and has the configuration illustrated in Figure 21. For example, terminal 30 includes a processor 311, memory 312, input / output interface 313, and communication interface 314, etc. The components of the processor 311, etc. are connected by an internal bus or the like and are configured to communicate with each other.

[0171] However, the configuration shown in Figure 21 is not intended to limit the hardware configuration of terminal 30. Terminal 30 may include hardware not shown, and may not have an input / output interface 313 if necessary. Also, the number of processors 311 etc. included in terminal 30 is not limited to the example in Figure 21; for example, terminal 30 may include multiple processors 311. Furthermore, terminal 30 only needs to be equipped with a camera for photographing the person being authenticated.

[0172] The processor 311 is a programmable device such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), or DSP (Digital Signal Processor). Alternatively, the processor 311 may be a device such as an FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit). The processor 311 executes various programs, including an operating system (OS).

[0173] Memory 312 includes RAM (Random Access Memory), ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), etc. Memory 312 stores the OS program, application programs, and various data.

[0174] The input / output interface 313 is an interface for a display device or input device (not shown). The display device is, for example, a liquid crystal display. The input device is, for example, a device that accepts user input such as a keyboard, mouse, or touch panel.

[0175] The communication interface 314 is a circuit, module, etc., that communicates with other devices. For example, the communication interface 314 includes a NIC (Network Interface Card), etc.

[0176] The functions of terminal 30 are realized by various processing modules. These processing modules are realized, for example, by the processor 311 executing a program stored in memory 312. The program can also be recorded on a computer-readable storage medium. The storage medium can be a non-transitory medium such as semiconductor memory, hard disk, magnetic recording medium, or optical recording medium. In other words, the present invention can also be embodied as a computer program product. Furthermore, the program can be downloaded via a network or updated using the storage medium on which the program is stored. Moreover, the processing module may be realized by a semiconductor chip.

[0177] The server device 10 can also be configured using an information processing device, similar to the terminal 30, and its basic hardware configuration is identical to that of the terminal 30, so its explanation will be omitted.

[0178] Terminal 30 is equipped with a computer, and its functions can be realized by having the computer execute a program. Furthermore, terminal 30 executes a control method for terminal 30 using this program. Similarly, server device 10 is equipped with a computer, and its functions can be realized by having the computer execute a program. Furthermore, server device 10 executes a control method for server device 10 using this program.

[0179] [Differentiation] The configuration and operation of the emotion analysis system described in the above embodiment are illustrative examples and are not intended to limit the system's configuration.

[0180] In the above embodiment, the operation of the emotion analysis system was described assuming a case where the same content is provided in the same place (same space, predetermined space), such as a movie theater or seminar venue. However, the emotion analysis system disclosed herein is also applicable to cases where the same content is provided in different locations, such as online seminars. In this case, each seminar participant simply needs to wear their own wearable terminal 20 and set up the terminal 30 so that it can capture images of their own face.

[0181] In the above embodiment, the emotion analysis system was explained using seminar participants as an example. However, the emotion analysis system may also analyze participants in other events. For example, movie viewers or conference attendees may be the subjects of emotion analysis.

[0182] The wearable device 20 may transmit vital data acquired from the user to the device 30 in real time, or it may transmit vital data acquired over a predetermined period to the device 30 all at once.

[0183] The wearable device 20 may acquire multiple types of vital data and transmit these multiple types of vital data to the terminal 30. For example, the wearable device 20 may transmit vital data such as pulse rate and body temperature, or pulse rate and blood pressure, to the terminal 30.

[0184] If multiple types of vital data are acquired, terminal 30 may calculate an emotion score using a learning model prepared for each type of vital data. Terminal 30 may also calculate the user's emotion score by performing arithmetic operations on the emotion score for each type of vital data.

[0185] Terminal 30 may calculate scores for each emotion. For example, terminal 30 may further subdivide positive emotions and calculate enjoyment scores for "fun" and excitement scores for "excitement." Alternatively, terminal 30 may further subdivide negative emotions and calculate sadness scores for "sadness," boredom scores for "boredom," and so on.

[0186] Terminal 30 may transmit subdivided emotion scores (e.g., enjoyment score, excitement score) to server device 10. Server device 10 may calculate analysis results for each emotion.

[0187] In the above embodiment, the case in which the emotion score is calculated in real time was described, but the terminal 30 may calculate the emotion score in batches at a predetermined timing. For example, the terminal 30 may calculate the emotion score in batches after the seminar has ended. Alternatively, the terminal 30 may calculate the emotion score after the seminar participant has given consent to provide their emotion information to a third party (for example, another seminar participant).

[0188] In the above embodiment, it was explained that emotional information (time-series data of emotional scores) is transmitted to the server device 10 after the seminar ends. However, emotional information (time-series data of emotional scores) may also be transmitted to the server device 10 in real time.

[0189] In the above embodiment, the case in which the wearable terminal 20 transmits vital data to terminal 30 was described. Each wearable terminal 20 distributed to seminar participants may centrally transmit vital data to a single device (vital data collection device). For example, the wearable terminal 20 obtains a user ID from terminal 30 and transmits the acquired vital data along with the user ID to the vital data collection device. The vital data collection device transmits the acquired vital data to terminal 30 or server device 10.

[0190] If the server device 10 can obtain attribute information (e.g., gender, age) for each seminar participant, it may calculate sentiment analysis results for each attribute. For example, the server device 10 can provide information such as the time periods that men found interesting and the time periods that women found interesting to the system administrator.

[0191] If the server device 10 can obtain video data of the seminar, it may provide the system administrator with the results of the sentiment analysis using the video data. For example, the server device 10 may present the analysis results to the system administrator by playing the video data on one screen (window) and displaying the sentiment score (average sentiment score of seminar participants) for the time period being played on a separate screen.

[0192] In the above embodiment, the case in which the participant management database is configured inside the server device 10 was described, but the database may be built on an external database server or the like. In other words, some functions of the server device 10 may be implemented in another device. More specifically, it is sufficient that the "emotion analysis unit (emotion analysis means)" etc. described above is implemented in any device included in the system.

[0193] The form of data transmission and reception between each device (terminal 30, server device 10) is not particularly limited, but the data transmitted and received between these devices may be encrypted. Since emotional information and the like are transmitted and received between these devices, it is desirable that encrypted data be transmitted and received in order to appropriately protect this information.

[0194] In the flowcharts (sequence diagrams) used in the above description, multiple processes (processes) are shown in order, but the execution order of the processes performed in the embodiment is not limited to the order in which they are shown. In the embodiment, the order of the illustrated processes can be changed to the extent that it does not impair the content, for example, by executing each process in parallel.

[0195] The embodiments described above are explained in detail to facilitate understanding of the disclosure, and it is not intended that all the configurations described above are necessary. Furthermore, when multiple embodiments are described, each embodiment may be used individually or in combination. For example, it is possible to replace parts of the configuration of one embodiment with those of another embodiment, or to add configurations from other embodiments to the configuration of one embodiment. In addition, it is possible to add, delete, or replace parts of the configuration of one embodiment with those of another.

[0196] As described above, the industrial applicability of the present invention is clear, and it is particularly suitable for use in emotion analysis systems and the like that which require accurate estimation of a user's emotions.

[0197] Some or all of the above embodiments may also be described as follows, but are not limited to the following: [Note 1] The same content is provided to multiple users, each using multiple devices they own. Server device, Includes, Each of the aforementioned multiple terminals is A vital data acquisition means for acquiring vital data of the wearer from a wearable device worn by the wearer, A face image acquisition means that acquires a face image of the owner by photographing the owner, A generation means generates emotional information relating to the emotions of the owner using the acquired vital data and the acquired facial image, and transmits an emotional information notification containing the generated emotional information to the server device. Equipped with, The server device is A receiving means for receiving the emotion information notification from each of the aforementioned multiple terminals, An analysis means analyzes the emotional information contained in each of the received emotional information notifications and generates an analysis result showing the emotional changes of all of the multiple users. A system equipped with these features. [Note 2] The same content is provided to multiple users, each using multiple devices they own. A camera device is installed to capture multiple users, A server device that stores the facial information of each of the aforementioned multiple users, Includes, The server device is Image data is acquired from the aforementioned camera device, For each of the multiple facial images captured in the aforementioned image data, the user corresponding to each facial image captured in the aforementioned image data is identified by a one-to-N matching (N is a positive integer) using the multiple facial images extracted from the aforementioned image data and the stored facial information. The facial image of the identified user is transmitted to the terminal of the identified user. Each of the aforementioned multiple terminals is The wearer's vital data is acquired from the wearable device being worn by the wearer. Using the acquired vital data and the facial image acquired from the server device, emotional information regarding the owner's emotions is generated. The emotion information notification containing the generated emotion information is transmitted to the server device. The server device is The emotion information notification is received from each of the aforementioned multiple terminals. A system that analyzes the emotional information contained in each of the received emotional information notifications and generates analysis results showing the emotional changes of all of the multiple users. [Note 3] The same content is provided to multiple users, each using multiple devices they own. A camera device is installed to capture multiple users, Server device, Includes, The server device is The camera device acquires image data, and transmits the acquired image data to each of the multiple terminals. Each of the aforementioned multiple terminals is It memorizes the owner's facial image, By performing a one-to-N matching (where N is a positive integer) using multiple facial images extracted from the aforementioned image data and the stored facial information of the owner, the facial image of the owner is identified from among the multiple facial images captured in the aforementioned image data. The vital data of the person being worn is acquired from the wearable device being worn by the person being worn. Using the acquired vital data and the identified facial image of the owner, emotional information regarding the owner's emotions is generated. The emotion information notification containing the generated emotion information is transmitted to the server device. The server device is The emotion information notification is received from each of the aforementioned multiple terminals. A system that analyzes the emotional information contained in each of the received emotional information notifications and generates analysis results showing the emotional changes of all of the multiple users. [Note 4] The same content is provided to multiple users, each using multiple devices they own. Server device, In a system including, Each of the aforementioned multiple terminals, The wearer's vital data is acquired from the wearable device being worn by the wearer. By photographing the aforementioned possessor, a facial image of the possessor is obtained. Using the acquired vital data and the acquired facial image, emotional information relating to the owner's emotions is generated, and an emotional information notification including the generated emotional information is transmitted to the server device. The aforementioned server device The emotion information notification is received from each of the aforementioned multiple terminals. A method for analyzing the emotional information contained in each of the received emotional information notifications and generating an analysis result that shows the emotional changes of all of the multiple users. [Note 5] A receiving means for receiving emotion information notifications from multiple devices, each owned by multiple users, to which the same content is provided. These notifications include emotion information about each user's emotions, generated using facial images obtained by each device capturing images of each user and vital data obtained from wearable devices worn by each user. An analysis means analyzes the emotional information contained in each of the received emotional information notifications and generates an analysis result showing the emotional changes of all of the multiple users. A server device equipped with the following features. [Note 6] The server device described in Appendix 5 generates the analysis result by calculating the average value of the emotion scores, which are numerical representations of the emotions of each of the multiple users, at a predetermined timing or over a predetermined period. [Note 7] On the computer installed in the terminal, A process to acquire vital data of the owner of the device from a wearable device worn by the owner of the device, A process to obtain a facial image of the owner by photographing the owner, A process to generate emotional information regarding the emotions of the possessor using the acquired vital data and the acquired facial image, The process of sending an emotion information notification containing the generated emotion information to a server device, A program to execute. [Note 8] The process of storing the facial information of the aforementioned owner, A process to perform a one-to-one matching using the stored facial information and the facial image obtained by the aforementioned photography, Let's execute it further, The process for generating the emotion information is the program described in Appendix 7, which generates the emotion information using the face image obtained by the photography that successfully performed the one-to-one matching. [Note 9] The process of transmitting the emotion information notification to the server device is the program described in Appendix 8, which transmits the emotion information notification to the server device when the owner's consent to transmit the emotion information to the server device is obtained. [Note 10] The process for generating the aforementioned emotion information is a program described in any one of the appendices 7 to 9, which inputs the acquired vital data and the acquired facial image into a learning model obtained by machine learning to obtain an emotion score that quantifies the emotions of the owner, and generates time-series data of the emotion score as the aforementioned emotion information.

[0198] Furthermore, each disclosure of the above-mentioned prior art documents cited herein is incorporated herein by reference. Although embodiments of the present invention have been described above, the present invention is not limited to these embodiments. It will be understood by those skilled in the art that these embodiments are merely illustrative and that various modifications are possible without departing from the scope and spirit of the present invention. That is, the present invention naturally includes the entire disclosure, including the claims, and various modifications and alterations that can be made by those skilled in the art in accordance with the technical idea. [Explanation of Symbols]

[0199] 10 Server devices 20 Wearable Devices 30 devices 40 Camera equipment 101 terminals 102 Server Equipment 111 Means of acquiring vital data 112. Method for acquiring facial images 113 Generation means 121 Receiving means 122 Analysis means 201 Communication Control Unit 202 Original Facial Image Management Department 203 Participant Registration Control Unit 204 Vital Data Acquisition Unit 205 Face Image Acquisition Unit 206 Emotion information generation section 207 Memory section 301 Communication Control Unit 302 Participant Management Department 303 Emotion Information Processing Unit 304 Emotion Analysis Department 305 Storage section 306 Participant Identification Department 307 Vital Data Processing Unit 308 Emotion information generation section 311 Processors 312 memory 313 Input / Output Interfaces 314 Communication Interface

Claims

1. The same content is provided to multiple users, each using multiple devices they own. Server device, Includes, Each of the aforementioned multiple terminals is A vital data acquisition means for acquiring vital data of the wearer from a wearable device worn by the wearer, A face image acquisition means that acquires a face image of the owner by photographing the owner, A generation means generates emotional information relating to the emotions of the owner using the acquired vital data and the acquired facial image, and transmits an emotional information notification containing the generated emotional information to the server device. Equipped with, The generation means inputs the acquired vital data and the acquired facial image into a learning model obtained by machine learning to obtain an emotion score that quantifies the emotions of the owner, and generates time-series data of the emotion score as emotion information. The server device is A receiving means for receiving the emotion information notification from each of the aforementioned multiple terminals, An analysis means analyzes the emotional information contained in each of the received emotional information notifications and generates an analysis result showing the emotional changes of all of the multiple users. A system equipped with these features.

2. The same content is provided to multiple users, each using multiple devices they own. A camera device is installed to capture multiple users, A server device that stores the facial information of each of the aforementioned multiple users, Includes, The server device is Image data is acquired from the aforementioned camera device, For each of the multiple facial images captured in the aforementioned image data, the user corresponding to each facial image captured in the aforementioned image data is identified by a one-to-N matching (N is a positive integer) using the multiple facial images extracted from the aforementioned image data and the stored facial information. The facial image of the identified user is transmitted to the terminal of the identified user. Each of the aforementioned multiple terminals is The wearer's vital data is acquired from the wearable device being worn by the wearer. By inputting the acquired vital data and the facial image acquired from the server device into a learning model obtained through machine learning, an emotion score that quantifies the owner's emotions is obtained, and time-series data of the emotion score is generated as emotion information related to the owner's emotions. The emotion information notification containing the generated emotion information is transmitted to the server device. The server device is The emotion information notification is received from each of the aforementioned multiple terminals. A system that analyzes the emotional information contained in each of the received emotional information notifications and generates analysis results showing the emotional changes of all of the multiple users.

3. The same content is provided to multiple users, each using multiple devices they own. A camera device is installed to capture multiple users, Server device, Includes, The server device is The camera device acquires image data, and transmits the acquired image data to each of the multiple terminals. Each of the aforementioned multiple terminals is It memorizes the owner's facial image, By performing a one-to-N matching (where N is a positive integer) using multiple facial images extracted from the aforementioned image data and the stored facial information of the owner, the facial image of the owner is identified from among the multiple facial images captured in the aforementioned image data. The vital data of the owner is acquired from the wearable device worn by the owner. By inputting the acquired vital data and the identified owner's facial image into a learning model obtained through machine learning, an emotion score that quantifies the owner's emotions is obtained, and time-series data of the emotion score is generated as emotion information related to the owner's emotions. The emotion information notification containing the generated emotion information is transmitted to the server device. The server device is The emotion information notification is received from each of the aforementioned multiple terminals. A system that analyzes the emotional information contained in each of the received emotional information notifications and generates analysis results showing the emotional changes of all of the multiple users.

4. The same content is provided to multiple users, each using multiple devices they own. Server device, In a system including, Each of the aforementioned multiple terminals, The wearer's vital data is acquired from the wearable device being worn by the wearer. By photographing the aforementioned possessor, a facial image of the possessor is obtained. By inputting the acquired vital data and the acquired facial image into a learning model obtained through machine learning, an emotion score that quantifies the owner's emotions is obtained, and time-series data of the emotion score is generated as emotion information regarding the owner's emotions. An emotion information notification including the generated emotion information is then transmitted to the server device. The aforementioned server device The emotion information notification is received from each of the aforementioned multiple terminals. A method for analyzing the emotional information contained in each of the received emotional information notifications and generating an analysis result that shows the emotional changes of all of the multiple users.

5. On the computer installed in the terminal, A process for acquiring vital data of the owner of the device from a wearable device worn by the owner of the device, A process to obtain a facial image of the owner by photographing the owner, A process to generate emotional information relating to the emotions of the possessor using the acquired vital data and the acquired facial image, The process of sending an emotion information notification containing the generated emotion information to a server device, A program to execute, The process for generating the aforementioned emotional information is a program that inputs the acquired vital data and the acquired facial image into a learning model obtained by machine learning to obtain an emotional score that quantifies the emotions of the owner, and generates time-series data of the emotional score as the aforementioned emotional information.

6. The process of storing the facial information of the aforementioned owner, A process to perform a one-to-one matching using the stored facial information and the facial image obtained by the aforementioned photography, Let's execute it further, The program according to claim 5, wherein the process for generating the emotion information generates the emotion information using the face image obtained by the photography that successfully performed the one-to-one matching.

7. The program according to claim 6, wherein the process of transmitting the emotion information notification to the server device is to transmit the emotion information notification to the server device when the consent of the owner to transmit the emotion information to the server device is obtained.

Images