Reaction sensing system and method for displaying reaction sensing result

Abstract

To prevent an individual from being identified when a state of a reaction shown by a participant in an online event is presented to a user, thereby allowing the participant to remain unaware that reactions are being sensed, and making it possible for the user to know the state of the reaction naturally shown by the participant.SOLUTION: A system comprises: a user terminal 2 having a camera 5 for imaging participants; and an analysis server 1 for acquiring, from the user terminal, a camera image obtained by imaging the participants, sensing reactions shown by the participants on the basis of the camera image of the participants, and generating display information by visualizing the sensing result. The analysis server aggregates reaction sensing results for each participant to generate reaction information (level information and proportion information) for all participants, and generates display information by visualizing the reaction information.SELECTED DRAWING: Figure 8

Description

Technical Field

[0005] ,

[0001] The present invention relates to a reaction detection system and a reaction detection result display method that detect reactions shown by participants in an online event and visualize and present the detection results.

Background Art

[0002] In recent years, online conference systems that hold conferences such as presentations online have become widespread. In an online conference system, camera images of the host and participants are displayed on the screen, and materials shared between the host (presenter) and participants are also displayed on the screen. However, there has been a problem that the reactions of participants to an online conference are not conveyed to the host and participants with a sense of presence as if they were on-site.

[0003] [ Therefore, as a technology for solving such problems in an online conference system, conventionally, there is known a technology in which participants are made to input impressions they felt about the content of an online conference, and the input content is aggregated and displayed (see Patent Document 1). Also, there is known a technology in which, by image recognition of a camera image of a participant, a nodding motion of the participant is detected, and information regarding the number and frequency of nodding motions for each participant is displayed on a terminal (see Patent Document 2).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] According to the technology described in Patent Document 1, since the participant themselves performs the input operation, it is not always possible to obtain information about the reactions that the participant actually showed. On the other hand, according to the technology described in Patent Document 2, since the reaction actions (nodding) shown by the participant are detected by image recognition of the camera image, information about the reactions that the participant actually showed can be obtained. However, with this technology, the status of the participant's reactions is displayed in a way that identifies the individual, so there is a problem in that participants become aware that reaction detection is being performed during the online event, and information about the reactions that the participant naturally shows cannot be obtained.

[0006] Therefore, the main objective of the present invention is to provide a reaction detection system and a reaction detection result display method that allow users to understand the reactions that participants naturally show, without the participants being aware that reaction detection is being performed, by presenting the status of reactions shown by participants in online events to the user in a way that does not identify individuals. [Means for solving the problem]

[0007] The present invention relates to a reaction detection system that detects reactions shown by participants in an online event and visualizes and presents the detection results. The system comprises a terminal device having a camera for photographing each participant, and a server device that acquires camera images of each participant from the terminal device, detects the reactions shown by each participant based on the camera images of each participant, and generates display information that visualizes the detection results. The server device aggregates the reaction detection results for each participant to generate reaction information for all participants, and generates the display information that visualizes this reaction information.

[0008] Furthermore, the reaction detection result display method of the present invention is a method for displaying reaction detection results that uses an information processing device to detect reactions shown by participants in an online event and visualize and present the detection results. The method acquires camera images of each participant from a terminal device having a camera that photographs each participant, detects the reactions shown by each participant based on the camera images of each participant, aggregates the reaction detection results for each participant to generate reaction information for all participants, and generates display information that visualizes that reaction information. [Effects of the Invention]

[0009] According to the present invention, display information that visualizes reaction information for all participants of an online event is presented to users such as participants. As a result, when the reaction status of online event participants is presented to users, individuals are not identified, so participants do not need to be aware that reaction detection is taking place, and users can understand the reaction status that participants are naturally showing. [Brief explanation of the drawing]

[0010] [Figure 1] Overall configuration diagram of the reaction detection system according to this embodiment [Figure 2] An explanatory diagram showing the presentation screen displayed on the participant's user terminal. [Figure 3] An explanatory diagram showing the presentation screen displayed on the participant's user terminal. [Figure 4] Diagram illustrating the post-event analysis results screen displayed on the organizer's user terminal. [Figure 5] Diagram illustrating the presentation screen displayed on the organizer's user terminal. [Figure 6] Diagram illustrating the presentation screen displayed on the organizer's user terminal. [Figure 7] Block diagram showing the general configuration of the analysis server. [Figure 8]Block diagram showing an overview of the processing performed on the analysis server. [Figure 9] This diagram illustrates the registration information for the document database and reaction database managed by the analysis server. [Figure 10] A flowchart illustrating the procedure for real-time reaction detection processing performed on the analysis server. [Figure 11] A flowchart illustrating the procedure for real-time reaction analysis processing performed on the analysis server. [Figure 12] A flowchart illustrating the procedure for post-analysis reaction processing performed on the analysis server. [Figure 13] A flowchart illustrating the procedure for information management processing related to the document database performed on the analysis server. [Modes for carrying out the invention]

[0011] The first invention made to solve the aforementioned problems is a reaction detection system that detects reactions shown by participants in an online event and visualizes and presents the detection results, comprising: a terminal device having a camera for photographing each participant; and a server device that acquires camera images of each participant from the terminal device, detects the reactions shown by each participant based on the camera images of each participant, and generates display information that visualizes the detection results, wherein the server device aggregates the reaction detection results for each participant to generate reaction information for all participants, and generates the display information that visualizes that reaction information.

[0012] According to this, a display of reaction information for all participants in an online event will be presented to participants and other users. This ensures that the reaction status of online event participants is not identified when presented to users, so participants do not need to be aware that reaction detection is taking place, and users can understand the reactions that participants are naturally showing.

[0013] Further, in the second invention, the server device is configured to generate the display information based on the real-time reaction detection results for each participant.

[0014] According to this, users such as participants can experience the atmosphere of the event with a sense of presence as if they were on the spot.

[0015] Further, in the third invention, the server device is configured to generate, as the reaction information, level information indicating the degree of positive or negative reactions for all participants.

[0016] According to this, users such as participants can easily grasp the degree of positive or negative reactions for all participants.

[0017] Further, in the fourth invention, the server device is configured to generate, as the reaction information, ratio information indicating the ratio for each reaction type for all participants.

[0018] According to this, users such as participants can easily grasp the ratio for each reaction type for all participants.

[0019] Further, in the fifth invention, after the online event ends, the server device is configured to generate, as the display information, a graph showing the change situation of reactions at each stage of the online event.

[0020] According to this, users such as participants can easily grasp the change situation of reactions at each stage of the online event. For example, when the online event is a presentation, a graph showing the change situation of reactions during the explanation period (display period) of each page of the presentation materials can be generated.

[0021] Furthermore, the sixth invention is configured such that the server device determines, as a reaction detection result, whether the reaction shown by the target participant is concentration, fatigue, agreement, dissatisfaction, or question.

[0022] According to this, it is possible to appropriately categorize the reactions shown by participants in online events.

[0023] Furthermore, the seventh invention is a reaction detection result display method in which an information processing device detects the reactions shown by participants in an online event and visualizes and presents the detection results, wherein the device acquires camera images of each participant from a terminal device having a camera that photographs each participant, detects the reactions shown by each participant based on the camera images of each participant, aggregates the reaction detection results for each participant to generate reaction information for all participants, and generates display information that visualizes that reaction information.

[0024] According to this, similar to the first invention, when presenting the status of reactions shown by participants in online events to the user, individuals are not identified. As a result, participants do not need to be aware that reaction detection is taking place, and the user can understand the status of reactions that participants are showing naturally.

[0025] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0026] Figure 1 is an overall diagram of the reaction detection system according to this embodiment.

[0027] This system detects the reactions of participants in online meetings (online events), visualizes the detection results, and presents them to participants and organizers. The system consists of an analysis server 1 (server device, information processing device) and user terminals 2 (terminal devices) operated by organizers and participants. The analysis server 1 is connected to the user terminals 2 via a network.

[0028] The analysis server 1 acquires camera images of the participants from the participants' user terminals 2, performs image analysis on those camera images to detect the reactions shown by the participants, and generates display information that visualizes the detection results. This display information is sent to the user terminal 2 and displayed on the screen.

[0029] User terminal 2 is equipped with a camera 5 for capturing images of participants and a display 6 for displaying screens streamed from analysis server 1. Participant user terminal 2 transmits camera images of participants captured by camera 5 to analysis server 1. Organizer (presenter) user terminal 2 also transmits camera images of the organizer captured by camera 5 to analysis server 1 as needed. User terminal 2 receives display information from analysis server 1 that visualizes the reaction detection results and displays it on the screen.

[0030] In this embodiment, the target online event is a meeting, particularly a presentation, but the target online event is not limited to meetings; for example, it could be a lecture, a training session, or an online class.

[0031] Next, we will explain the presentation screens 101 and 111 that are displayed on the participant's user terminal 2. Figures 2 and 3 are explanatory diagrams showing the presentation screens 101 and 111 that are displayed on the participant's user terminal 2.

[0032] On participant user terminal 2, presentation screens 101 and 111 (see Figures 2 and 3) are displayed during the online meeting (presentation).

[0033] Presentation screens 101 and 111 are equipped with a document display unit 102. The document display unit 102 displays the presentation materials. The presentation materials are uploaded in advance from the presenter's user terminal 2 to the analysis server 1 and distributed to each participant's user terminal 2. As a result, the presentation materials are displayed on the presenter's user terminal 2 and each participant's user terminal 2, and the presentation materials are shared between the presenter and each participant. The analysis server 1 may also work in conjunction with an online meeting management server (not shown) or an online meeting application installed on the user terminal 2 to distribute the presentation screens 101 and 111 to users.

[0034] Furthermore, presentation screens 101 and 111 are equipped with a level button 103 and a ratio button 104. When a participant operates the level button 103, the presentation screen 101 (see Figure 2) switches to the level display mode. When a participant operates the ratio button 104, the presentation screen 111 (see Figure 3) switches to the ratio display mode.

[0035] Furthermore, presentation screens 101 and 111 are equipped with an analysis results display unit 105. The analysis results display unit 105 displays the reaction status of all participants in real time. The content displayed by the analysis results display unit 105 differs depending on the display mode (level display mode and ratio display mode).

[0036] Furthermore, the display content of the analysis results display unit 105 is updated in accordance with the real-time reaction detection and reaction analysis processes, which are executed at predetermined intervals (e.g., every 1 second, 3 seconds, 5 seconds, etc.). As a result, the display content of the analysis results display unit 105 changes moment by moment in accordance with the progress of the presentation. Therefore, participants can grasp the reaction status of all participants at a glance and experience the atmosphere of the event with a sense of presence as if they were actually there.

[0037] In the presentation screen 101 (see Figure 2) in level display mode, the analysis results display unit 105 displays an image 106 representing the reaction level (level information), which indicates the degree of positive reactions among all participants. The reaction level is expressed as a percentage of the participants who showed a positive reaction. In the image 106 representing the reaction level, multiple bars are drawn side by side, and the bar corresponding to the height of the reaction level is lit up. The bars are displayed in color, and for example, they may be displayed in gray, blue, green, yellow, and red from lowest to highest level.

[0038] Participants can understand how many participants are showing positive reactions based on the reaction level. For example, a high reaction level indicates that many participants are showing positive reactions, meaning the online meeting is engaging. On the other hand, a low reaction level indicates that few participants are showing positive reactions, meaning the online meeting is not engaging. In this case, participants should try to liven up the online meeting by asking questions or taking other actions.

[0039] On the other hand, in the presentation screen 111 in ratio display mode (see Figure 3), the analysis results display unit 105 displays an image 112 representing the reaction ratio (ratio information), which shows the ratio of each reaction type for all participants. In the reaction ratio image 112, the ratio for each reaction type is represented by a 100% stacked bar graph. The reaction types are concentration, fatigue, agreement, dissatisfaction, and question, and the ratio for each reaction type is drawn in a different color.

[0040] Participants can understand which reactions are most common by tracking reaction ratios. For example, if many participants show reactions indicating concentration or fatigue, they should try to revitalize the online meeting by asking questions or engaging in casual conversation.

[0041] By the way, presentation screens 101 and 111 (see Figures 2 and 3) are displayed on the participant's user terminal 2, but a similar screen may also be displayed on the organizer's (presenter's) user terminal 2. This allows the organizer to understand how many participants are showing positive reactions based on their reaction levels, and to understand what state most participants are in based on their reaction ratios. In this case, the organizer (presenter) should try to liven up the online meeting by changing the flow of the meeting, asking questions, or engaging in casual conversation.

[0042] In this embodiment, the presentation screen 101 (see Figure 2) in level display mode displays a reaction level representing the percentage of participants who showed a positive reaction to all participants. Here, a positive reaction is when the reaction type is concentration and agreement. On the other hand, the reaction level may also represent the percentage of participants who showed a negative reaction to all participants. A negative reaction is when the reaction type is fatigue, dissatisfaction, and questioning. In addition, the reaction level for each reaction type (concentration, fatigue, agreement, dissatisfaction, and questioning) specified by the user (participant or organizer) from the reaction level setting screen (not shown) may be displayed.

[0043] Furthermore, in this embodiment, in the presentation screen 111 (see Figure 3) in ratio display mode, the reaction ratio is shown as a ratio for each type of reaction (concentration, fatigue, agreement, dissatisfaction, and question). However, reactions that do not fall under either positive or negative reactions may be considered neutral reactions, and the reaction ratio may be shown as a ratio for each of positive, negative, and neutral reactions.

[0044] Next, we will explain the post-event analysis results screen 121 displayed on the organizer's user terminal 2. Figure 4 is an explanatory diagram showing the post-event analysis results screen 121. Note that the post-event analysis results screen 121 may also be displayed on the participants' user terminal 2.

[0045] On the organizer's user terminal 2, the post-meeting analysis results screen 121 is displayed after the online meeting (presentation) has ended.

[0046] The post-analysis results screen 121 displays a bar graph 122 and a line graph 123 representing the results of the post-meeting reaction analysis process. The bar graph 122 and line graph 123 show how participants' reactions changed at each stage of the online meeting. Specifically, the explanation period for each page of the presentation materials corresponds to each stage of the online meeting (presentation), and the bar graph 122 and line graph 123 show how participants reacted while the presenter was explaining each page of the presentation materials.

[0047] Bar graph 122, similar to the reaction ratio image 112 displayed in the analysis results display unit 105 of the presentation screen 111 (see Figure 3) in ratio display mode, shows the ratio of each reaction type for all participants on a page-by-page basis using a 100% stacked bar graph. This allows users (such as organizers) to understand what kinds of reactions are most common among participants as the presenter explains each page of the presentation materials.

[0048] The line graph 123, similar to the reaction level image 106 displayed in the analysis results display unit 105 of the presentation screen 101 (see Figure 2) in level display mode, shows the reaction level on a page-by-page basis, representing the percentage of participants who showed a positive reaction among all participants. This allows users (such as presenters) to understand how many participants showed a positive reaction as the presenter explained each page of the presentation materials. Alternatively, the line graph 123 may also represent the percentage of participants who showed a negative reaction among all participants.

[0049] Users (such as presenters) can view the post-event analysis results screen 121 to understand the participants' reactions to each page after the event, which is something that cannot be obtained through surveys conducted with participants. In addition, users (such as presenters) can identify pages where participants did not react well and check for problems with the content or explanations on those pages.

[0050] Next, we will explain the presentation screens 131 and 141 displayed on the presenter's (organizer's) user terminal 2. Figures 5 and 6 are explanatory diagrams showing the presentation screens 131 and 141 displayed on the presenter's (organizer's) user terminal 2.

[0051] On the presenter's (organizer's) user terminal 2, the presentation screen 131 (initial screen) shown in Figure 5 is displayed when the presentation begins.

[0052] The presentation screen 131 shown in Figure 5 is equipped with a document display unit 132. The document display unit 132 displays the presentation materials that have been set to be shared with the participants.

[0053] Furthermore, the presentation screen 131 is provided with a presentation ID input section 133 and a presenter name input section 134. In the presentation ID input section 133, the presenter can input a presentation ID by typing text or selecting from a pull-down menu. In the presenter name input section 134, the presenter can input their name by typing text or selecting from a pull-down menu. Note that the presentation ID may be automatically assigned when the presentation materials are registered with the analysis server 1 and notified to the presenter in advance.

[0054] Furthermore, the presentation screen 131 is equipped with a start button 135 and an end button 136. The presenter operates the start button 135 when starting the presentation. The presenter also operates the end button 136 when ending the presentation. This allows the analysis server 1 to obtain the duration of the presentation (start time and end time).

[0055] Furthermore, when the presenter operates the start button 135 on presentation screen 131 (see Figure 5), the presentation screen transitions to presentation screen 141 shown in Figure 6.

[0056] In the presentation screen 141 shown in Figure 6, page forward buttons 142 and page back buttons 143 are displayed on the document display unit 132. When the presenter operates the page forward button 142, the presentation materials displayed on the document display unit 132 move to the next page. Conversely, when the presenter operates the page back button 143, the presentation materials displayed on the document display unit 132 return to the previous page. The analysis server 1 can obtain the page display period (start time and end time) by acquiring the time when the page forward button 142 and the page back button 143 are operated.

[0057] Next, we will describe the general configuration of analysis server 1. Figure 7 is a block diagram showing the general configuration of analysis server 1. Figure 8 is a block diagram showing an overview of the processing performed by analysis server 1.

[0058] The analysis server 1 comprises a communication unit 11, a storage unit 12, and a processor 13.

[0059] The communication unit 11 communicates with the organizer's user terminal 2 and the participants' user terminals 2 via the network.

[0060] The storage unit 12 stores programs and other data executed by the processor 13. The storage unit 12 also stores registration information for the document database and the reaction database (see Figure 9).

[0061] The processor 13 performs various processes by executing programs stored in the memory unit 12. In this embodiment, the processor 13 performs real-time reaction detection processing P1, real-time reaction analysis processing P2, post-reaction reaction analysis processing P3, and information management processing P4, etc.

[0062] In the real-time reaction detection process P1, the processor 13 acquires camera images of each participant from the user terminal 2 and detects the reactions shown by each participant in real time based on each participant's camera image. The real-time reaction detection process P1 also includes facial expression detection process P11, gaze detection process P12, motion recognition process P13, and integrated judgment process P14.

[0063] In the facial expression detection process P11, the processor 13 detects the facial expressions of the participants based on the camera image.

[0064] In the gaze detection process P12, the processor 13 detects the participant's gaze based on the camera image.

[0065] In the motion recognition process P13, the processor 13 recognizes the participant's movements based on the camera image.

[0066] In the integrated judgment process P14, the processor 13 determines, based on the detection results of the facial expression detection process P11, the detection results of the gaze detection process P12, and the recognition results of the motion recognition process P13, which type of reaction the target participant is showing as a reaction detection result: concentration, fatigue, agreement, dissatisfaction, or questioning.

[0067] Here, for example, the facial expression detected by the facial expression detection process P11 is used to estimate whether the participant's emotion is in one of three states: positive, negative, or neutral. Based on this estimation, the type of reaction, such as agreement or dissatisfaction, is determined.

[0068] Furthermore, the participant's level of concentration is estimated from the movement of their eyes detected by the eye-tracking process P12. If the participant's gaze detected by the eye-tracking process P12 is directed towards the display screen of user terminal 2, it is estimated that the participant is intently watching the display screen. On the other hand, if the participant's gaze is not directed towards the display screen, it is estimated that the participant is looking away. Based on these estimation results, the type of reaction, such as concentration or fatigue, is determined.

[0069] Furthermore, the type of fatigue reaction is determined from the participant's body movements and posture detected by motion recognition processing P13. Additionally, the type of agreement reaction is determined from the participant's nodding motion detected by motion recognition processing P13. Furthermore, the type of questioning reaction is determined from the participant's head tilting motion detected by motion recognition processing P13.

[0070] In this embodiment, the reaction detection result determines whether the reaction shown by the target participant is of one of the following types: concentration, fatigue, agreement, dissatisfaction, or question. However, other reaction types may also be set. For example, if it is difficult to set a reaction to one of the five types, a no-response reaction type may be set.

[0071] In the real-time reaction analysis process P2, the processor 13 obtains the latest reaction detection results (reaction type) for each participant acquired in the reaction detection process P1, aggregates the latest reaction detection results for each participant, generates reaction information for all participants, and generates display information that visualizes that reaction information.

[0072] Specifically, the processor 13 generates level information (reaction level) regarding the percentage of participants who showed a positive reaction to all participants, and generates display information that visualizes this level information. Here, the level information is expressed as a percentage of the total number of participants, representing the number of participants who fall under the category of positive reaction. The display information is an image 106 representing the reaction level displayed on the analysis result display unit 105 of the presentation screen 101 (see Figure 2) in level display mode.

[0073] Furthermore, the processor 13 generates ratio information regarding the proportion of participants for each reaction type and generates display information that visualizes this ratio information. Here, the ratio information is expressed as a percentage, with the total number of participants for all participants being 100 percent, and the ratio of the number of participants for each reaction type being expressed as a percentage. The display information is an image 112 representing the reaction ratios displayed on the analysis result display unit 105 of the presentation screen 111 (see Figure 3) in ratio display mode.

[0074] In the post-event reaction analysis process P3, the processor 13 retrieves the reaction detection results for each participant related to the target online meeting (presentation) at each time point from the information obtained in the reaction detection process P1 and registered in the reaction DB, aggregates the reaction detection results for each participant at each time point, generates reaction information for each stage of the online meeting, and generates display information that visualizes the reaction information for each stage.

[0075] In this scenario, if the target online meeting (online event) is a presentation, the duration for which the presenter explains each page of the presentation materials—that is, the display period of each page (from the start time to the end time)—corresponds to each stage of the online meeting. The display period of each page is retrieved from the document database. Additionally, a graph showing the changes in reactions for each page is generated as display information.

[0076] Specifically, the processor 13 generates level information for each period of page display, specifically information on the percentage of participants who showed a positive reaction to all participants, and generates display information that visualizes the level information for each period of page display. The display information is a line graph 123 that is displayed on the post-analysis results screen 121 (see Figure 4) on the participant's user terminal 2.

[0077] Furthermore, the processor 13 generates ratio information for each page display period, specifically information regarding the ratio of participants for each reaction type, and generates display information that visualizes the ratio information for each page display period. The display information is a bar graph 122 that is displayed on the post-analysis results screen 121 (see Figure 4) on the participant's user terminal 2.

[0078] Furthermore, when aggregating the reaction detection results for each participant at each time point to generate reaction information for each stage of an online meeting, reaction information for one stage can be obtained by statistically processing (e.g., averaging) the reaction information (level information and ratio information) for each time point included in the display period of one page.

[0079] In information management processing P4, processor 13 manages a document database where information about documents (presentation materials) used in online meetings is registered. In addition, in information management processing, processor 13 manages a reaction database where information about reactions shown by participants in online meetings (presentations) is registered. The registration information for the document database and the reaction database is stored in the storage unit 12.

[0080] Furthermore, facial expression detection, gaze detection, and motion recognition may be performed using machine learning models (image recognition engines) constructed using machine learning methods such as deep learning. Similarly, result integration may also be performed using machine learning models constructed using machine learning methods such as deep learning.

[0081] Next, we will explain the registration information for the document database and the reaction database managed by analysis server 1. Figure 9 is an explanatory diagram showing the registration information for the document database and the reaction database.

[0082] On analysis server 1, documents shared between presenters (organizers) and participants during online meetings (presentations), specifically information about presentation materials, are managed by a document database.

[0083] As shown in Figure 9(A), the document database registers the document ID (presentation ID) of the presentation materials, the total number of pages in the presentation materials, and the name of the presenter. In addition, the document database registers information about each page of the presentation materials, such as the page number and the display period of the page (start time and end time).

[0084] Furthermore, on analysis server 1, information regarding the reactions shown by participants during online meetings is managed by the reaction database.

[0085] As shown in Figure 9(B), the reaction database registers information about the target online meeting (presentation), such as the document ID (presentation ID) and page number. The reaction database also registers information about the reaction detection process, such as the participant's registration ID, the time the reaction detection process was executed, and the reaction detection result (reaction type).

[0086] Next, we will explain the procedure for real-time reaction detection processing performed on analysis server 1. Figure 10 is a flowchart showing the procedure for real-time reaction detection processing.

[0087] Analysis server 1 performs real-time reaction detection processing. During this time, reaction detection processing is repeatedly performed at each time point for all participants in the online meeting (presentation) while the meeting is taking place.

[0088] Specifically, analysis server 1 repeatedly executes the processing related to the first reaction detection loop (ST101~ST110). The processing related to the first reaction detection loop is repeated periodically during the target period, specifically until the end of this online meeting (presentation).

[0089] In the first reaction detection, the analysis server 1 repeatedly executes the processes (ST102~ST109) related to the second reaction detection loop. The processes related to the second reaction detection loop are performed for all participants in the online meeting and are repeated until processing for all participants in the online meeting is completed.

[0090] In the second reaction detection loop, the analysis server 1 first acquires camera images from the target participant's user terminal 2 (ST103). Alternatively, the user terminal 2 may detect the participant's head from the camera image and extract a partial image corresponding to the face, which the analysis server 1 then acquires. In this case, the amount of data of the acquired images can be reduced, thereby suppressing the load on the network and allowing subsequent processing to be carried out efficiently.

[0091] Next, the analysis server 1 detects the facial expressions of the target participants based on the camera images (facial expression detection process) (ST104).

[0092] Furthermore, the analysis server 1 detects the gaze of the target participant based on the camera image (gaze detection processing) (ST105).

[0093] Furthermore, the analysis server 1 recognizes the actions of the target participant based on the camera images (action recognition processing) (ST106).

[0094] Next, the analysis server 1 determines the type of reaction of the target participant (one of the following: concentration, fatigue, agreement, dissatisfaction, or questioning) as a reaction detection result based on the detection results of the facial expression detection process, the detection results of the gaze detection process, and the recognition results of the motion recognition process (integrated determination process) (ST107).

[0095] Next, the analysis server 1 saves the reaction detection results (reaction type) of the target participant to the storage unit 12 along with the current time (ST108).

[0096] The processing related to the first and second reaction detection loops described above is repeated until the target period, i.e., the end of the online meeting (presentation), so that the reaction detection results of each participant at each time point in the target period are accumulated in the storage unit 12.

[0097] Next, we will describe the procedure for the real-time reaction analysis process performed on analysis server 1. Figure 11 is a flowchart showing the procedure for the real-time reaction analysis process.

[0098] When the analysis server 1 receives a request for real-time reaction analysis results from a participant's user terminal 2 (ST201), it starts processing related to the reaction analysis loop (ST202~ST210). The reaction analysis loop is repeated at predetermined intervals until the requests from user terminal 2 are finished.

[0099] In the reaction analysis loop, first, analysis server 1 obtains the latest reaction detection results (reaction type) for each participant (ST203). Next, it determines whether the information requested from user terminal 2 is level information or ratio information (ST204).

[0100] If the information requested from user terminal 2 is level information (referred to as "level information" in ST204), analysis server 1 aggregates the latest reaction detection results (reaction type) for each participant and generates the latest level information, specifically information regarding the percentage of participants who showed a positive reaction for all participants (ST205). Next, analysis server 1 generates display information that visualizes the latest level information (ST206).

[0101] On the other hand, if the information requested from user terminal 2 is ratio information (referred to as "ratio information" in ST204), analysis server 1 aggregates the latest reaction detection results (reaction types) for each participant and generates the latest ratio information, specifically information regarding the ratio of participants for each reaction type (ST207). Next, analysis server 1 generates display information that visualizes the latest ratio information (ST208).

[0102] Next, the analysis server 1 transmits the information requested by the user terminal 2, namely, display information that visualizes the latest level information or ratio information, to the user terminal 2 (ST209). As a result, on the user terminal 2, an image 106 representing the reaction level is displayed on the analysis result display unit 105 of the presentation screen 101 (see Figure 2), or an image 112 representing the reaction ratio is displayed on the analysis result display unit 105 of the presentation screen 111 (see Figure 3).

[0103] Next, we will explain the procedure for the post-event reaction analysis process performed on analysis server 1. Figure 12 is a flowchart showing the procedure for the post-event reaction analysis process.

[0104] When the analysis server 1 receives a request for post-event reaction analysis results from a participant's user terminal 2 (ST301), it retrieves the reaction detection results for each participant related to the target presentation (online meeting) at each time point from the reaction DB (see Figure 9(B)) (ST302).

[0105] Next, the analysis server 1 aggregates the reaction detection results for each participant at each time point and generates level information (reaction information) for each page display period (from start time to end time), specifically information on the percentage of participants who showed a positive reaction to all participants (ST303).

[0106] Furthermore, the analysis server 1 aggregates the reaction detection results for each participant at each time point to generate ratio information (reaction information) for each page display period, specifically information on the ratio of participants for each type of reaction (concentration, fatigue, agreement, dissatisfaction, and question) (ST304).

[0107] Next, analysis server 1 generates display information that visualizes level information for each period of page display, and display information that visualizes ratio information for each period of page display (ST305).

[0108] Next, the analysis server 1 transmits display information regarding level information and ratio information to the user terminal 2 (ST306). As a result, the user terminal 2 displays a bar graph 122 and a line graph 123 on the post-analysis results screen 121 (see Figure 4).

[0109] Next, we will explain the procedure for information management processing related to the document database performed on analysis server 1. Figure 13 is a flowchart showing the procedure for information management processing related to the document database.

[0110] Analysis Server 1 manages a document database (see Figure 9(A)) where information about documents used in online meetings, specifically presentation materials, is registered (information management processing). On the presenter's (organizer's) user terminal 2, on presentation screens 131 and 141 (see Figures 5 and 6), the presenter performs operations such as entering the document ID (presentation ID) and the presenter's name, starting the presentation, navigating to other pages, and ending the presentation. Analysis Server 1 performs information management processing based on notifications sent from User Terminal 2 in response to the presenter's operations.

[0111] Specifically, when the analysis server 1 receives a presentation start notification from the user terminal 2 (ST401), it sets the time of that notification as the start time of the first page of the presentation materials and registers a new document related to this presentation in the document database (ST402).

[0112] Next, analysis server 1 starts processing related to the information management loop (ST403~ST406). Processing related to the information management loop will be repeated periodically until this online meeting (presentation) ends.

[0113] In the information management loop, when the analysis server 1 receives a page transition notification from the user terminal 2 (ST404), it sets the time at that moment as the end time of the source page and the start time of the destination page, and updates the registration information in the document database related to the current presentation (ST405).

[0114] The information management loop process will be repeated until the presentation is finished, specifically until a presentation completion notification is received from user terminal 2.

[0115] Next, the analysis server 1 sets the time it receives the presentation end notification from user terminal 2 as the end time of the last page and updates the registration information in the document database related to this presentation (ST407).

[0116] Furthermore, as a reaction detection system, the analysis server 1 acquires camera images of participants from the participants' user terminals 2, performs image analysis on those camera images, and detects the reactions shown by the participants. Alternatively, the camera images of participants may be taken on the participants' user terminals 2, where image analysis is performed on those camera images, and the reaction detection results shown by the participants are sent to the server device and stored in the reaction database.

[0117] As described above, embodiments have been explained as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these embodiments and can be applied to embodiments that have been modified, replaced, added, or omitted. Furthermore, it is possible to create new embodiments by combining the components described in the above embodiments. [Industrial applicability]

[0118] The reaction detection system and reaction detection result display method according to the present invention have the effect of allowing users to understand the reactions that participants naturally show, without identifying individuals when presenting the status of reactions shown by participants in an online event, so that participants do not have to be aware that reaction detection is being performed. Therefore, the reaction detection system and reaction detection result display method are useful as a system that detects the reactions shown by participants in an online event and visualizes and presents the detection results. [Explanation of symbols]

[0119] 1. Analysis Server (Server device, information processing device) 2. User terminal (terminal device) 5 Cameras 6 displays 11 Communications Department 12 Storage section 13 processors

Claims

1. A reaction detection system that detects reactions shown by participants of an online event and visualizes and presents the detection results, A terminal device having a camera for photographing each participant; a server device that acquires camera images of each participant from the terminal device, detects which of a plurality of reaction types each participant shows based on the camera images of each participant, and generates analysis result display information that visualizes the reaction status of all participants; The server device includes: A reaction detection system characterized by setting, as display modes for the analysis result display information targeted at all participants, a first mode that displays level information indicating the degree to which a specific reaction type was detected, and a second mode that displays ratio information indicating the ratio of each of the detected reaction types, and generating an analysis result display screen having an operation unit that transitions to either the first mode or the second mode.

2. The server device includes:

2. The reaction detection system according to claim 1, wherein the analysis result display information is generated based on a detection result of the reaction type for each participant in real time.

3. The server device includes:

3. The reaction detection system according to claim 2, wherein level information indicating the degree of positive or negative reaction for all participants is generated as the analysis result display information.

4. The server device includes: The reaction detection system according to claim 1, further comprising: generating, as the analysis result display information, a graph showing changes in reactions at each stage of the online event after the online event ends.

5. The server device includes:

2. The reaction detection system according to claim 1, wherein the reaction type is determined to be one of concentration, fatigue, satisfaction, dissatisfaction, and doubt shown by the target participant.

6. A reaction detection result display method for detecting reactions shown by participants of an online event and visualizing and presenting the detection results by an information processing device, comprising: Acquire camera images of each participant from a terminal device having a camera for photographing each participant; Based on the camera images of each participant, it is detected which of a number of reaction types the reaction shown by each participant corresponds to, and analysis result display information is generated that visualizes the reaction status of all participants; A reaction detection result display method characterized by setting, as display modes for the analysis result display information targeted at all participants, a first mode that displays level information indicating the degree to which a specified reaction type was detected, and a second mode that displays ratio information indicating the ratio of each of the detected reaction types, and generating an analysis result display screen having an operation unit that transitions to either the first mode or the second mode.

Images