Reaction detection system and reaction detection result display method

Abstract

To prevent an individual from being identified when a state of a reaction shown by a participant in an online event is presented to a user, thereby allowing the participant to remain unaware that reactions are being sensed, and making it possible for the user to know the state of the reaction naturally shown by the participant.SOLUTION: A system comprises: a user terminal 2 having a camera 5 for imaging participants; and an analysis server 1 for acquiring, from the user terminal, a camera image obtained by imaging the participants, sensing reactions shown by the participants on the basis of the camera image of the participants, and generating display information by visualizing the sensing result. The analysis server aggregates reaction sensing results for each participant to generate reaction information (level information and proportion information) for all participants, and generates display information by visualizing the reaction information.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to a reaction detection system and a reaction detection result display method that detects reactions shown by participants in an online event and visualizes and presents the detection results. [Background technology]

[0002] In recent years, online conference systems for holding online meetings such as presentations have become popular. In online conference systems, camera images of the host and participants are displayed on the screen, as well as materials shared between the host (presenter) and participants. However, there has been a problem in that participants' reactions to online conferences are not conveyed to the host and participants in a realistic manner, as if they were actually there.

[0003] To solve these problems in online conference systems, a known technique involves having participants input their impressions of the content of an online conference, and then aggregating and displaying the inputs (see Patent Document 1). Another known technique involves detecting participants' nodding movements through image recognition of camera images of the participants, and displaying information on the number and frequency of nodding movements for each participant on a terminal (see Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-36640 [Patent Document 2] Japanese Patent Application Laid-Open No. 2009-267621 Summary of the Invention [Problem to be solved by the invention]

[0005] According to the technology of Patent Document 1, participants themselves perform the input operation, so information on the reactions actually shown by the participants is not necessarily obtained. On the other hand, according to the technology of Patent Document 2, reaction movements (nodding movements) shown by participants are detected by image recognition of camera images, so information on the reactions actually shown by the participants is obtained. However, with this technology, the status of participants' reactions is displayed in a manner that identifies individuals, so there is a problem that during the online event, participants become aware that reaction detection is being performed, and information on the reactions naturally shown by the participants cannot be obtained.

[0006] Therefore, the main objective of the present invention is to provide a reaction detection system and a reaction detection result display method that allows users to understand the reaction situations naturally shown by participants in an online event by not identifying individuals when presenting the reaction situations shown by participants to users, so that participants do not need to be aware that reaction detection is being performed. [Means for solving the problem]

[0007] The reaction detection system of the present invention detects reactions shown by participants in an online event and visualizes and presents the detection results. The reaction detection system includes a terminal device having a camera that photographs each participant, and acquires camera images of each participant from the terminal device, and visualizes and presents the reactions shown by each participant based on the camera images of each participant. Detects which of several reaction types a reaction falls into and displays the analysis results that visualize the reaction status of all participants. and a server device that generates the As a display mode of the analysis result display information for all participants, a first mode displays level information indicating the degree to which a predetermined reaction type is detected, and a second mode displays level information indicating the degree to which a predetermined reaction type is detected. The aforementioned a second mode for displaying ratio information representing the ratio of each reaction type; The aforementioned an analysis result display screen having an operation unit for transitioning to either the first mode or the second mode; The configuration is to generate the following.

[0008] The reaction detection result display method of the present invention is a reaction detection result display method in which reactions shown by participants in an online event are detected and the detection results are visualized and presented by an information processing device, and the method acquires camera images of each participant from a terminal device having a camera that photographs each participant, and visualizes and presents the reactions shown by each participant based on the camera images of each participant. Detects which of several reaction types a reaction falls into and displays the analysis results that visualize the reaction status of all participants. Generate an analysis result display screen having an operation unit for switching between a first mode that displays level information indicating the degree to which a predetermined reaction type was detected and a second mode that displays ratio information indicating the ratio of each of the detected reaction types, as display modes of the analysis result display information targeted at all participants; and The configuration is to generate the following. [Effects of the Invention]

[0009] According to the present invention, a method for collecting information from all participants of an online event is Two display modes are set for the analysis result display information that visualizes the reaction situation, and both modes can be switched. This allows participants to see the reactions of participants in online events without being aware that they are being detected, as individuals are not identified when the reactions are presented to users. In multiple display modes It can be grasped. [Brief explanation of the drawings]

[0010] [Figure 1] Overall configuration diagram of a reaction detection system according to this embodiment [Figure 2] An explanatory diagram showing a presentation screen displayed on a participant's user terminal. [Figure 3] An explanatory diagram showing a presentation screen displayed on a participant's user terminal. [Figure 4] An explanatory diagram showing the post-analysis result screen displayed on the organizer's user terminal. [Figure 5] FIG. 10 is an explanatory diagram showing a presentation screen displayed on the host's user terminal. [Figure 6] FIG. 10 is an explanatory diagram showing a presentation screen displayed on the host's user terminal. [Figure 7] Block diagram showing the general configuration of the analysis server [Figure 8] Block diagram showing the overview of the processing performed by the analysis server [Figure 9]An explanatory diagram showing the registered information of the document DB and the registered information of the reaction DB managed by the analysis server. [Figure 10] Flow diagram showing the real-time reaction detection process performed on the analysis server [Figure 11] Flow diagram showing the procedure for real-time reaction analysis processing performed on the analysis server [Figure 12] Flow diagram showing the procedure for post-event reaction analysis processing performed on the analysis server [Figure 13] Flow diagram showing the procedure for managing information about the document database performed by the analysis server DETAILED DESCRIPTION OF THE INVENTION

[0011] The first invention made to solve the above problem is a reaction detection system that detects reactions shown by participants in an online event and visualizes and presents the detection results, and includes a terminal device having a camera that photographs each participant, and a server device that acquires camera images of each participant from the terminal device, detects reactions shown by each participant based on the camera images of each participant, and generates display information that visualizes the detection results, and the server device is configured to aggregate the reaction detection results for each participant to generate reaction information targeted at all participants, and generate the display information that visualizes the reaction information.

[0012] According to this, display information that visualizes reaction information for all participants in an online event is presented to users such as participants. As a result, individuals are not identified when the reaction status shown by participants in an online event is presented to users, so participants do not need to be aware that reaction detection is being performed, and users can understand the reaction status shown naturally by participants.

[0013] In a second aspect of the present invention, the server device generates the display information based on the reaction detection result of each participant in real time.

[0014] This allows users, such as participants, to experience the atmosphere of the event with a sense of realism as if they were actually there.

[0015] In a third aspect of the present invention, the server device generates, as the reaction information, level information that indicates the degree of a positive or negative reaction of all participants.

[0016] This allows users such as participants to easily understand the degree of positive or negative reactions from all participants.

[0017] In a fourth aspect of the present invention, the server device generates, as the reaction information, ratio information that indicates a ratio of each reaction type among all participants.

[0018] This allows participants and other users to easily grasp the ratio of each reaction type across all participants.

[0019] In a fifth aspect of the present invention, the server device generates, after the online event ends, a graph representing changes in reactions at each stage of the online event as the display information.

[0020] This allows participants and other users to easily understand how reactions change at each stage of an online event. For example, if the online event is a presentation, a graph can be generated that shows how reactions change during the explanation period (display period) of each page of the presentation materials.

[0021] In a sixth aspect of the present invention, the server device is configured to determine, as the reaction detection result, whether the reaction shown by the target participant is concentration, fatigue, understanding, dissatisfaction, or doubt.

[0022] This allows us to appropriately categorize the reactions of participants in online events.

[0023] Furthermore, the seventh invention is a reaction detection result display method in which an information processing device detects reactions shown by participants in an online event and visualizes and presents the detection results. The method is configured to obtain camera images of each participant from a terminal device having a camera that photographs each participant, detect reactions shown by each participant based on the camera images of each participant, aggregate the reaction detection results for each participant to generate reaction information targeted at all participants, and generate display information that visualizes the reaction information.

[0024] According to this, as with the first invention, individuals are not identified when the reaction status of participants in an online event is presented to users, so participants do not need to be aware that reaction detection is taking place, and users can understand the reaction status that participants naturally show.

[0025] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0026] FIG. 1 is a diagram showing the overall configuration of a reaction detection system according to this embodiment.

[0027] This system detects reactions shown by participants in an online conference (online event), visualizes the detection results, and presents them to the participants and organizers. The system includes an analysis server 1 (server device, information processing device) and user terminals 2 (terminal devices) operated by the organizer and participants. The analysis server 1 is connected to the user terminals 2 via a network.

[0028] The analysis server 1 acquires camera images of the participants from the participants' user terminals 2, performs image analysis on the camera images, detects reactions shown by the participants, and generates display information that visualizes the detection results. This display information is sent to the user terminals 2 and displayed on the screen.

[0029] The user terminal 2 is equipped with a camera 5 that captures images of the participants and a display 6 that displays the screen distributed from the analysis server 1. The user terminal 2 of the participant transmits camera images of the participants captured by the camera 5 to the analysis server 1. The user terminal 2 of the organizer (presenter) also transmits camera images of the organizer captured by the camera 5 to the analysis server 1 as needed. The user terminal 2 receives display information that visualizes the reaction detection results from the analysis server 1 and displays it on the screen.

[0030] In this embodiment, the target online event is a conference, particularly a presentation, but the target online event is not limited to a conference and may be, for example, a lecture, a seminar, an online class, etc.

[0031] Next, a description will be given of the presentation screens 101 and 111 displayed on the user terminals 2 of the participants. Figures 2 and 3 are explanatory diagrams showing the presentation screens 101 and 111 displayed on the user terminals 2 of the participants.

[0032] On the user terminals 2 of the participants, presentation screens 101 and 111 (see FIGS. 2 and 3) are displayed during the online conference (presentation).

[0033] The presentation screens 101, 111 are provided with a document display unit 102. Presentation materials are displayed on the document display unit 102. The presentation materials are uploaded in advance from the user terminal 2 of the organizer (presenter) to the analysis server 1 and distributed to the user terminals 2 of each participant. As a result, the presentation materials are displayed on the user terminal 2 of the organizer and the user terminals 2 of each participant, and the presentation materials are shared between the organizer and each participant. The analysis server 1 may distribute the presentation screens 101, 111 to users in cooperation with an online conference management server (not shown) or an online conference application installed on the user terminal 2.

[0034] Furthermore, presentation screens 101 and 111 are provided with a level button 103 and a ratio button 104. When a participant operates level button 103, the presentation screen transitions to presentation screen 101 in level display mode (see FIG. 2). When a participant operates ratio button 104, the presentation screen transitions to presentation screen 111 in ratio display mode (see FIG. 3).

[0035] Furthermore, the presentation screens 101 and 111 are provided with an analysis result display section 105. The analysis result display section 105 displays the reaction status of all participants in real time. The display content of the analysis result display section 105 differs depending on the display mode (level display mode and ratio display mode).

[0036] Furthermore, the display content of analysis result display unit 105 is updated as real-time reaction detection processing and reaction analysis processing are executed at predetermined intervals (for example, every 1 second, 3 seconds, 5 seconds, etc.). As a result, the display content of analysis result display unit 105 changes from moment to moment as the presentation progresses. This allows participants to grasp the state of reactions of all participants at a glance, and to experience the atmosphere of the event with a sense of realism as if they were actually there.

[0037] On the presentation screen 101 in the level display mode (see FIG. 2), an image 106 showing a reaction level (level information) indicating the degree of positive reaction among all participants is displayed in the analysis result display section 105. The reaction level is expressed as a percentage of the participants who showed a positive reaction. In the image 106 showing the reaction level, multiple bars are drawn side by side, and the bars corresponding to the height of the reaction level are displayed lit up. The bars may be displayed in color, for example, gray, blue, green, yellow, and red, in order of decreasing level.

[0038] The reaction level allows participants to understand how many participants are showing positive reactions. Here, for example, if the reaction level is high, participants can understand that many participants are showing positive reactions, i.e., the online conference is lively. On the other hand, if the reaction level is low, participants can understand that few participants are showing positive reactions, i.e., the online conference is not lively. In this case, participants should try to liven up the online conference by asking questions, etc.

[0039] On the other hand, on the presentation screen 111 in the ratio display mode (see FIG. 3), an image 112 showing reaction ratios (ratio information) that indicate the ratios of each reaction type for all participants is displayed in the analysis result display section 105. In the reaction ratio image 112, the ratios of each reaction type are shown as a 100% stacked bar graph. The reaction types include concentration, fatigue, satisfaction, dissatisfaction, and doubt, and the ratios of each reaction type are drawn in different colors.

[0040] Participants can understand what type of reactions are most prevalent among participants based on the reaction ratio. For example, if many participants show reactions of concentration or fatigue, participants should try to liven up the online meeting by asking questions or chatting.

[0041] Incidentally, while presentation screens 101, 111 (see FIGS. 2 and 3) are displayed on the user terminals 2 of the participants, similar screens may also be displayed on the user terminal 2 of the organizer (presenter). This allows the organizer to understand how many participants are showing positive reactions based on the reaction levels, and what state most of the participants are in based on the reaction ratio. In this case, the organizer (presenter) may try to liven up the online conference by changing the course of the online conference, asking questions, or engaging in casual conversation.

[0042] In this embodiment, the presentation screen 101 (see FIG. 2) in the level display mode displays a reaction level that indicates the percentage of participants who showed a positive reaction to all participants. Here, a positive reaction refers to a case where the reaction type is concentration and understanding. On the other hand, the reaction level may also indicate the percentage of participants who showed a negative reaction to all participants. A negative reaction refers to a case where the reaction type is fatigue, dissatisfaction, or doubt. Furthermore, a reaction level may be displayed for each reaction type (concentration, fatigue, understanding, dissatisfaction, and doubt) specified by a user (participant or organizer) on a reaction level setting screen (not shown).

[0043] Furthermore, in this embodiment, on the presentation screen 111 in the ratio display mode (see FIG. 3), the reaction ratio is expressed as a ratio for each reaction type (concentration, fatigue, satisfaction, dissatisfaction, and doubt), but a reaction that does not fall into either a positive or negative reaction may be considered a neutral reaction, and the reaction ratio may be expressed as a ratio for each of positive, negative, and neutral.

[0044] Next, a description will be given of the post-analysis result screen 121 displayed on the organizer's user terminal 2. Fig. 4 is an explanatory diagram showing the post-analysis result screen 121. Note that the post-analysis result screen 121 may also be displayed on the user terminals 2 of the participants.

[0045] After the online conference (presentation) ends, the post-analysis result screen 121 is displayed on the user terminal 2 of the organizer.

[0046] The post-analysis result screen 121 displays a bar graph 122 and a line graph 123 that represent the results of the post-analysis reaction analysis process. The bar graph 122 and the line graph 123 represent changes in the reactions of the participants at each stage of the online conference. Specifically, the explanation period for each page of the presentation materials corresponds to each stage of the online conference (presentation), and the bar graph 122 and the line graph 123 represent how the participants reacted when the presenter explained each page of the presentation materials.

[0047] The bar graph 122, like the reaction ratio image 112 displayed in the analysis result display section 105 of the presentation screen 111 (see FIG. 3) in ratio display mode, shows the ratio of each reaction type for all participants as a 100% stacked bar graph on a page-by-page basis. This allows the user (such as the organizer) to understand what type of reaction is most common among participants when the presenter explains each page of the presentation materials.

[0048] Similar to the reaction level image 106 displayed in the analysis result display section 105 of the presentation screen 101 (see FIG. 2) in level display mode, the line graph 123 shows the reaction level, which indicates the percentage of participants who showed positive reactions, for each page of all participants. This allows the user (such as the presenter) to understand how many participants showed positive reactions when the presenter explains each page of the presentation materials. The line graph 123 may also show the percentage of participants who showed negative reactions, for all participants.

[0049] By viewing the post-analysis result screen 121, users (presenters, etc.) can grasp the participants' reactions to each page after the event, which is something that cannot be grasped through a questionnaire administered to participants. In addition, users (presenters, etc.) can identify pages that did not receive a good reaction from participants and check for problems with the descriptions and explanations on those pages.

[0050] Next, a description will be given of presentation screens 131 and 141 displayed on the user terminal 2 of the presenter (host). Figures 5 and 6 are explanatory diagrams showing presentation screens 131 and 141 displayed on the user terminal 2 of the presenter (host).

[0051] At the start of a presentation, a presentation screen 131 (initial screen) shown in FIG. 5 is displayed on the user terminal 2 of the presenter (host).

[0052] 5 is provided with a document display section 132. The document display section 132 displays presentation materials that are set to be shared among the participants.

[0053] The presentation screen 131 also has a presentation ID input section 133 and a presenter name input section 134. In the presentation ID input section 133, the presenter can input a presentation ID by entering characters or selecting from a pull-down menu. In the presenter name input section 134, the presenter can enter their own name by entering characters or selecting from a pull-down menu. Note that the presentation ID may be automatically assigned when the presentation materials are registered in the analysis server 1, and may be notified to the presenter in advance.

[0054] The presentation screen 131 is also provided with a start button 135 and an end button 136. The presenter operates the start button 135 when starting a presentation. The presenter also operates the end button 136 when ending the presentation. This allows the analysis server 1 to obtain the duration of the presentation (start time and end time).

[0055] Furthermore, when the presenter operates the start button 135 on the presentation screen 131 (see FIG. 5), the screen transitions to a presentation screen 141 shown in FIG.

[0056] 6, a page forward button 142 and a page back button 143 are displayed in the document display section 132. When the presenter operates the page forward button 142, the presentation material displayed in the document display section 132 transitions to the next page. On the other hand, when the presenter operates the page back button 143, the presentation material displayed in the document display section 132 returns to the previous page. The analysis server 1 can obtain the display period (start time and end time) of the page by obtaining the times when the page forward button 142 and the page back button 143 are operated.

[0057] Next, we will explain the general configuration of the analysis server 1. Fig. 7 is a block diagram showing the general configuration of the analysis server 1. Fig. 8 is a block diagram showing an overview of the processing performed by the analysis server 1.

[0058] The analysis server 1 includes a communication unit 11, a storage unit 12, and a processor 13.

[0059] The communication unit 11 communicates with the user terminal 2 of the organizer and the user terminals 2 of the participants via the network.

[0060] The storage unit 12 stores programs and the like to be executed by the processor 13. The storage unit 12 also stores registered information in the document DB and registered information in the reaction DB (see FIG. 9).

[0061] The processor 13 performs various processes by executing programs stored in the storage unit 12. In this embodiment, the processor 13 performs a real-time reaction detection process P1, a real-time reaction analysis process P2, a post-event reaction analysis process P3, an information management process P4, and the like.

[0062] In the real-time reaction detection process P1, the processor 13 acquires camera images of each participant from the user terminal 2 and detects the reactions of each participant in real time based on the camera images of each participant. In the real-time reaction detection process P1, a facial expression detection process P11, a gaze detection process P12, a motion recognition process P13, and an integrated judgment process P14 are performed.

[0063] In the facial expression detection process P11, the processor 13 detects the facial expressions of the participants based on the camera images.

[0064] In the gaze detection process P12, the processor 13 detects the gaze of the participants based on the camera image.

[0065] In the action recognition process P13, the processor 13 recognizes the actions of the participants based on the camera images.

[0066] In the integrated determination process P14, the processor 13 determines, based on the detection results of the facial expression detection process P11, the detection results of the gaze detection process P12, and the recognition results of the action recognition process P13, whether the reaction shown by the target participant is a reaction type of concentration, fatigue, satisfaction, dissatisfaction, or doubt, as a reaction detection result.

[0067] Here, for example, from the facial expressions detected in the facial expression detection process P11, it is estimated whether the participant's emotion is positive, negative, or neutral, and based on this estimation result, the reaction type, such as satisfaction or dissatisfaction, is determined.

[0068] Furthermore, the concentration level of the participant is estimated from the movement of the participant's gaze detected in the gaze detection process P12. Furthermore, if the gaze of the participant detected in the gaze detection process P12 is directed toward the display screen of the user terminal 2, it is estimated that the participant is gazing at the display screen. On the other hand, if the gaze of the participant is not directed toward the display screen, it is estimated that the participant is looking away. Based on these estimation results, the reaction type, such as concentration or fatigue, is determined.

[0069] Furthermore, the reaction type of fatigue is determined from the participant's body movements and posture detected in the action recognition process P13. Furthermore, the reaction type of understanding is determined from the participant's nodding action detected in the action recognition process P13. Furthermore, the reaction type of questioning is determined from the participant's tilting of the head detected in the action recognition process P13.

[0070] In this embodiment, the reaction detection result is determined as the reaction type of the target participant, which is concentration, fatigue, understanding, dissatisfaction, or doubt, but other reaction types may be set. For example, if it is difficult to set one of the five reaction types, a reaction type of no reaction may be set.

[0071] In the real-time reaction analysis process P2, the processor 13 acquires the latest reaction detection results (reaction type) for each participant acquired in the reaction detection process P1, aggregates the latest reaction detection results for each participant, generates reaction information for all participants, and generates display information that visualizes the reaction information.

[0072] Specifically, the processor 13 generates level information (reaction level) regarding the proportion of participants who showed positive reactions to all participants, and generates display information that visualizes the level information. Here, the level information is expressed as a percentage, which is the proportion of participants who fall into the positive reaction type relative to the total number of participants. The display information is an image 106 that indicates the reaction level and is displayed in the analysis result display section 105 of the presentation screen 101 (see FIG. 2) in the level display mode.

[0073] The processor 13 also generates ratio information regarding the ratio of participants for each reaction type, and generates display information that visualizes the ratio information. Here, the ratio information indicates the ratio of the number of participants for each reaction type as a percentage, with the total number of participants being 100 percent. The display information is an image 112 that represents the reaction ratios and is displayed in the analysis result display section 105 of the presentation screen 111 (see FIG. 3) in the ratio display mode.

[0074] In the post-event reaction analysis process P3, the processor 13 obtains the reaction detection results for each time of each participant involved in the target online conference (presentation) from the information obtained in the reaction detection process P1 and registered in the reaction DB, aggregates the reaction detection results for each time of each participant, generates reaction information for each stage of the online conference, and generates display information that visualizes the reaction information for each stage.

[0075] If the target online conference (online event) is a presentation, the period during which the presenter explains each page of the presentation materials, i.e., the display period of each page (the time from the start time to the end time), corresponds to each stage of the online conference. The display period of each page is obtained from the document database. A graph showing the change in reaction status for each page is also generated as display information.

[0076] Specifically, the processor 13 generates level information for each display period of the page, specifically, information regarding the percentage of participants who showed positive reactions to all participants, and generates display information that visualizes the level information for each display period of the page. The display information is a line graph 123 that is displayed on the post-analysis result screen 121 (see FIG. 4) on the user terminal 2 of the participant.

[0077] The processor 13 also generates ratio information for each display period of the page, specifically, information on the ratio of participants for each reaction type, and generates display information that visualizes the ratio information for each display period of the page. The display information is a bar graph 122 that is displayed on the post-analysis result screen 121 (see FIG. 4) on the user terminal 2 of the participant.

[0078] When aggregating the reaction detection results for each participant at each time and generating reaction information for each stage of an online conference, the reaction information for one stage can be obtained by statistically processing (e.g., averaging) the reaction information (level information and ratio information) for each time included in one stage, specifically the display period of one page.

[0079] In the information management process P4, the processor 13 manages a document DB in which information about documents (presentation materials) used in the online conference is registered. Also, in the information management process, the processor 13 manages a reaction DB in which information about reactions shown by participants in the online conference (presentation) is registered. The registered information in the document DB and the registered information in the reaction DB are accumulated in the memory unit 12.

[0080] The facial expression detection process, the gaze detection process, and the action recognition process may be performed using a machine learning model (image recognition engine) constructed by machine learning such as deep learning. The result integration process may also be performed using a machine learning model constructed by machine learning such as deep learning.

[0081] Next, a description will be given of the registered information in the document DB and the registered information in the reaction DB managed by the analysis server 1. Fig. 9 is an explanatory diagram showing the registered information in the document DB and the registered information in the reaction DB.

[0082] In the analysis server 1, documents shared between a presenter (host) and participants in an online conference (presentation), specifically information about presentation materials, are managed in a document DB.

[0083] 9(A), the document DB stores the document ID (presentation ID) of the presentation materials, the total number of pages in the presentation materials, the name of the presenter, etc. The document DB also stores information about each page of the presentation materials, such as the page number and the display period (start time and end time) of the page.

[0084] Furthermore, in the analysis server 1, information on reactions shown by participants in an online conference is managed in a reaction DB.

[0085] 9(B), the reaction DB stores information about the target online conference (presentation), such as the document ID (presentation ID), page number, etc. The reaction DB also stores information about the reaction detection process, such as the reception ID of the target participant, the time when the reaction detection process was executed, and the reaction detection result (reaction type).

[0086] Next, we will explain the procedure of the real-time reaction detection process carried out by the analysis server 1. Fig. 10 is a flow diagram showing the procedure of the real-time reaction detection process.

[0087] The analysis server 1 executes a real-time reaction detection process. At this time, the reaction detection process is repeatedly executed at each time for all participants in the online conference while the online conference (presentation) is being held.

[0088] Specifically, the analysis server 1 repeatedly executes the processing (ST101 to ST110) related to the first reaction detection loop. The processing related to the first reaction detection loop is periodically repeated until the target period, specifically, the current online conference (presentation) ends.

[0089] In the first reaction detection, the analysis server 1 repeatedly executes the processing (ST102 to ST109) related to the second reaction detection loop. The processing related to the second reaction detection loop is performed for all participants in the online conference, and is repeated until the processing for all participants in the online conference is completed.

[0090] In the second reaction detection loop, the analysis server 1 first acquires a camera image from the user terminal 2 of the target participant (ST103). Here, the user terminal 2 may detect the participant's head from the camera image and extract a partial image corresponding to the facial image, which is then acquired by the analysis server 1. In this case, the amount of data of the acquired image can be reduced, suppressing the load on the network and enabling subsequent processing to be performed efficiently.

[0091] Next, the analysis server 1 detects the facial expressions of the target participants based on the camera image (facial expression detection process) (ST104).

[0092] Furthermore, the analysis server 1 detects the gaze of the target participant based on the camera image (gaze detection process) (ST105).

[0093] Furthermore, the analysis server 1 recognizes the actions of the target participants based on the camera images (action recognition process) (ST106).

[0094] Next, the analysis server 1 determines the reaction type of the target participant (one of concentration, fatigue, satisfaction, dissatisfaction, and doubt) as the reaction detection result based on the detection results of the facial expression detection process, the detection results of the gaze detection process, and the recognition results of the action recognition process (integrated determination process) (ST107).

[0095] Next, the analysis server 1 stores the reaction detection results (reaction types) of the target participants together with the current time in the storage unit 12 (ST108).

[0096] The processing related to the first and second reaction detection loops described above is repeated until the target period, i.e., the current online conference (presentation), ends, and the reaction detection results of each participant at each time during the target period are accumulated in memory unit 12.

[0097] Next, we will explain the procedure of the real-time reaction analysis process performed by the analysis server 1. Fig. 11 is a flowchart showing the procedure of the real-time reaction analysis process.

[0098] When the analysis server 1 receives a request for real-time reaction analysis results from a participant's user terminal 2 (ST201), it starts processing related to a reaction analysis loop (ST202 to ST210). The reaction analysis loop is repeated at a predetermined cycle until the request from the user terminal 2 is completed.

[0099] In the reaction analysis loop, first, the analysis server 1 acquires the latest reaction detection result (reaction type) of each participant (ST203), and then determines whether the information requested by the user terminal 2 is level information or ratio information (ST204).

[0100] Here, if the information requested from the user terminal 2 is level information ("Level Information" in ST204), the analysis server 1 aggregates the latest reaction detection results (reaction types) of each participant to generate the latest level information, specifically, information on the proportion of participants who showed positive reactions to all participants (ST205). Next, the analysis server 1 generates display information that visualizes the latest level information (ST206).

[0101] On the other hand, if the information requested by the user terminal 2 is ratio information ("Ratio Information" in ST204), the analysis server 1 aggregates the latest reaction detection results (reaction types) of each participant to generate the latest ratio information, specifically, information on the ratio of participants for each reaction type (ST207). Next, the analysis server 1 generates display information that visualizes the latest ratio information (ST208).

[0102] Next, the analysis server 1 transmits the information requested by the user terminal 2, i.e., display information visualizing the latest level information or ratio information, to the user terminal 2 (ST209). As a result, on the user terminal 2, an image 106 representing the reaction level is displayed in the analysis result display section 105 of the presentation screen 101 (see FIG. 2), or an image 112 representing the reaction ratio is displayed in the analysis result display section 105 of the presentation screen 111 (see FIG. 3).

[0103] Next, a description will be given of the procedure of post-event reaction analysis processing performed by the analysis server 1. Fig. 12 is a flow diagram showing the procedure of post-event reaction analysis processing.

[0104] When the analysis server 1 receives a request for post-event reaction analysis results from a participant's user terminal 2 (ST301), it obtains the reaction detection results for each participant involved in the presentation (online conference) at each time from the reaction DB (see Figure 9(B)) (ST302).

[0105] Next, the analysis server 1 aggregates the reaction detection results for each participant at each time and generates level information (reaction information) for each page display period (from the start time to the end time), specifically, information on the percentage of participants who showed positive reactions to all participants (ST303).

[0106] In addition, the analysis server 1 aggregates the reaction detection results for each participant at each time and generates ratio information (reaction information) for each page display period, specifically, information on the ratio of participants for each reaction type (concentrated, tired, satisfied, dissatisfied, and questioning) (ST304).

[0107] Next, the analysis server 1 generates display information that visualizes the level information for each display period of the page, and display information that visualizes the ratio information for each display period of the page (ST305).

[0108] Next, the analysis server 1 transmits display information relating to the level information and ratio information to the user terminal 2 (ST306). As a result, the user terminal 2 displays a bar graph 122 and a line graph 123 on the post-analysis result screen 121 (see FIG. 4).

[0109] Next, a description will be given of the procedure for information management processing relating to the document DB, which is carried out by the analysis server 1. Fig. 13 is a flowchart showing the procedure for information management processing relating to the document DB.

[0110] The analysis server 1 manages a document DB (see FIG. 9(A)) in which documents used in online conferences, specifically information about presentation materials, are registered (information management process). On the user terminal 2 of the presenter (organizer), the presenter performs operations on presentation screens 131, 141 (see FIGS. 5 and 6) to input a document ID (presentation ID) and the presenter's name, start the presentation, transition to a new page, and end the presentation. The analysis server 1 performs information management process based on notifications sent from the user terminal 2 in response to the presenter's operations.

[0111] Specifically, when the analysis server 1 first receives a presentation start notification from the user terminal 2 (ST401), it sets the current time as the start time of the first page of the presentation materials and registers a new document related to this presentation in the document DB (ST402).

[0112] Next, the analysis server 1 starts the processing related to the information management loop (ST403 to ST406). The processing related to the information management loop is periodically repeated until the current online conference (presentation) ends.

[0113] In the information management loop, when the analysis server 1 receives a page transition notification from the user terminal 2 (ST404), it sets the time at that time as the end time of the page from which the transition originates and the start time of the page to which the transition is to be made, and updates the registration information in the document DB related to this presentation (ST405).

[0114] The process related to the information management loop is repeated until the current presentation ends, specifically, until a presentation end notification is received from the user terminal 2.

[0115] Next, the analysis server 1 sets the time when it receives the presentation end notification from the user terminal 2 as the end time of the last page, and updates the registration information in the document DB relating to this presentation (ST407).

[0116] In addition, as a reaction detection system, the analysis server 1 acquires camera images of the participants from the participants' user terminals 2, and performs image analysis processing on the camera images to detect reactions shown by the participants. Alternatively, the analysis server 1 may perform image analysis processing on the camera images of the participants in the participants' user terminals 2, and send the reaction detection results shown by the participants to the server device and store them in the reaction DB.

[0117] As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. Furthermore, it is also possible to combine the components described in the above embodiments to create new embodiments. [Industrial Applicability]

[0118] The reaction detection system and reaction detection result display method of the present invention have the advantage that, by not identifying individuals when presenting the reaction status of participants in an online event to users, participants are not aware that reaction detection is taking place and users can grasp the reaction status that participants naturally show.The system is useful as a reaction detection system and reaction detection result display method that detects reactions shown by participants in an online event and visualizes and presents the detection results. [Explanation of symbols]

[0119] 1. Analysis server (server device, information processing device) 2. User terminal (terminal device) 5. Camera 6. Display 11 Communications Department 12 Storage section 13 processors

Claims

1. A reaction detection system that detects reactions shown by participants in an online event and visualizes and presents the detection results, a terminal device having a camera for photographing each participant; a server device that acquires camera images of each participant from the terminal device, detects which of a plurality of reaction types the reaction shown by each participant corresponds to based on the camera image of each participant, and generates analysis result display information that visualizes the reaction status of all participants; The server device A reaction detection system characterized by setting two display modes for the analysis result display information for all participants: a first mode that displays level information indicating the degree to which a specified reaction type was detected, and a second mode that displays ratio information indicating the ratio of each detected reaction type, and generating an analysis result display screen having an operation unit that transitions to either the first mode or the second mode.

2. The server device The reaction detection system according to claim 1 , wherein the analysis result display information is generated based on the reaction type detection results for each participant in real time.

3. The server device 3. The reaction detection system according to claim 2, wherein level information indicating the degree of positive or negative reaction for all participants is generated as the analysis result display information.

4. The server device The reaction detection system according to claim 1, further comprising: generating a graph showing changes in reactions at each stage of the online event as the analysis result display information after the online event has ended.

5. The server device The reaction detection system according to claim 1, wherein the reaction type is determined to be one of concentration, fatigue, satisfaction, dissatisfaction, and doubt shown by the target participant.

6. A reaction detection result display method for detecting reactions shown by participants of an online event and visualizing and presenting the detection results using an information processing device, Acquire camera images of each participant from a terminal device having a camera for photographing each participant; Based on the camera images of each participant, the system detects which of multiple reaction types each participant's reaction corresponds to, and generates analysis result display information that visualizes the reaction status of all participants. A reaction detection result display method characterized by setting two display modes for the analysis result display information for all participants: a first mode that displays level information indicating the degree to which a specified reaction type was detected, and a second mode that displays ratio information indicating the ratio of each detected reaction type, and generating an analysis result display screen having an operation unit that transitions to either the first mode or the second mode.

Images