Information processing system and information processing method

Abstract

To provide a technique for increasing a conference activity level.SOLUTION: An information processing system 1 acquires a face image of a conference participant, acquires a feature quantity of a gesture and a pulse rate of the conference participant from the face image, estimates an aspect of the conference participant based on the feature quantity of the gesture and the pulse rate, and displays the estimated aspect of the conference participant in a display.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system and an information processing method. [Background technology]

[0002] In recent years, the number of opportunities to hold online meetings via networks has increased. In online meetings, participants' facial expressions and actions cannot be seen, making communication more difficult than in face-to-face meetings.

[0003] As a technology for supporting the progress of such online meetings, Patent Document 1 describes a meeting support device that detects the emotional expressions of meeting attendees, predicts whether the attendee intends to speak based on the emotional expressions, and asks the attendee for their opinion if it is predicted that the attendee intends to speak.

[0004] Furthermore, Patent Document 2 describes a meeting support device that acquires biometric information of meeting participants, calculates an activity level that indicates the degree of activity of discussion in the meeting based on the acquired biometric information, recognizes the timing for a facilitator to intervene and encourage participants to speak, and once the timing for intervention is recognized, outputs activity level information that indicates changes in the activity level based on the activity level calculated within a specified period thereafter. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2022-139436 A [Patent Document 2] Patent Publication No. 2022-68904 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in Patent Document 1, if it is predicted that an attendee has the intention to speak, the attendee is asked for their opinion, but since the opinion is asked without any human judgment, it is difficult to control the frequency and order of questions.

[0007] In addition, in Patent Document 2, when the personal feelings of a participant who has not spoken for a long time change, the facilitator is notified that it is time to intervene with the participant. However, there are many situations in which it is better to judge the need for intervention as a factor independent of whether or not the participant has spoken, and further improvement is needed.

[0008] The present invention has been made to solve the above-mentioned problems, and has an object to provide a technique that can improve the liveliness of a meeting. [Means for solving the problem]

[0009] In order to solve the above problems, one representative information processing system of the present invention includes an imaging unit that acquires facial images of conference participants, a facial image analysis processing unit that acquires gesture features and pulse rates of the conference participants from the facial images, a behavior analysis processing unit that estimates the appearance of the conference participants based on the gesture features and pulse rates, and a display unit that displays the estimated appearance of the conference participants. Effect of the Invention

[0010] According to the present invention, it is possible to improve the liveliness of a meeting.

[0011] Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram illustrating an example of a configuration of an information processing system according to a first embodiment. [Diagram 2] FIG. 2 illustrates an example of a configuration of a terminal. [Diagram 3] FIG. 2 illustrates an example of a server configuration. [Figure 4] 11 is a flowchart illustrating an example of information acquisition processing according to the first embodiment. [Diagram 5] 13 is a flowchart illustrating an example of a gesture analysis process. [Figure 6] 13 is a flowchart showing an example of details of a pulse rate estimation process. [Figure 7] 13 is a flowchart illustrating an example of a manner analysis process. [Figure 8] FIG. 1 is a diagram for explaining mode analysis. [Figure 9] FIG. 13 is a diagram showing an example of a display screen shown to participants. [Figure 10] 13 is a flowchart illustrating an example of a mode determination process. [Figure 11] FIG. 13 is a diagram showing an example of a display screen shown to participants. [Figure 12] FIG. 11 is a block diagram illustrating an example of a configuration of an information processing system according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the embodiments will be described with reference to the drawings. EXAMPLES

[0014] Hereinafter, a first embodiment of the present invention will be described with reference to FIGS.

[0015] FIG. 1 is a block diagram illustrating an example of a configuration of an information processing system according to a first embodiment.

[0016] 1, an information processing system 1 is, for example, a conference support system for smoothly operating a conference, and includes terminals 2 used by participants, and a server 3. These devices are connected via a network to perform data communication.

[0017] The terminal 2 executes an input process 21 that processes input data from connected devices such as a camera and a microphone, a transmission / reception process 22 that controls data transmission and reception via the network, a display process 23 that displays information on the screen, and the like.

[0018] The server 3 executes a facial image analysis process 31 that analyzes facial movement and blood flow data from image data transmitted from the terminal 2, a voice analysis process 32 that analyzes voice data transmitted from the terminal 2, a transmission / reception process 33 that controls data transmission and reception via the network, and a behavior analysis process 34 that estimates the behavior of participants from the processing results of the facial image analysis process 31 and the voice analysis process 32.

[0019] Basically, each participant in a conference has one terminal 2, and multiple terminals 2 are connected to one server. Each terminal 2 corresponds to a personal computer, tablet terminal, smartphone, etc. The terminal 2 acquires video and audio data of each participant in the conference and transfers the data to the server 3.

[0020] The server 3 analyzes the data sent from the terminal 2 and calculates the attitude (attitude / reaction, such as positive / negative or understanding / doubting) of each participant toward the topic being discussed during the meeting. The calculation results are displayed on the screens of the meeting participants, especially the facilitator, showing who has which attitude / reaction, making it possible to encourage speaking up and to use this information in planning the management of the meeting. This ultimately makes it possible to support smooth management of the meeting and invigorate discussion.

[0021] The terminal 2 and server 3 include the necessary functions and performance to achieve the above-mentioned conference support, which will be described in detail later. The communication method between the terminal 2 and server 3 can be realized by using a general server / client communication function.

[0022] FIG. 2 is a diagram illustrating an example of the configuration of the terminal 2. As shown in FIG.

[0023] Terminal 2 may be a so-called personal computer or tablet terminal, and comprises a calculation unit (e.g., a Central Processing Unit) 201, a memory 202, an imaging unit 203 consisting of a visible light camera that captures the faces of participants and serves as an information input means, a microphone 204 that detects the voice emitted by the user, a communication unit 205 that transmits and receives data via a network, a memory unit 206 for storing information, such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), and a display unit 210 that displays information on, for example, a display, all of which are connected by a system bus.

[0024] The calculation unit 201 implements necessary functions by executing a program stored in the storage unit 206 .

[0025] The memory 202 is a main storage device used when the calculation unit 201 executes processing, and is composed of a volatile storage element such as a RAM (Random Access Memory).

[0026] The communication unit 205 is an interface used by the terminal 2 to communicate with an external device, and is configured with a NIC (Network Interface Card) etc. The communication unit 205 is connected to a network (e.g., the Internet) and communicates with the external device via the network.

[0027] Processing programs are stored in the storage unit 206, and when executed, they are expanded in the memory 202 and operate. The processing programs include programs that realize an input processing unit 207 that processes input data from connected devices such as a camera and a microphone, a transmission / reception processing unit 208 that controls data transmission and reception via a network, and a display processing unit 209 that displays information on a screen.

[0028] Note that the terminal 2 may further include components other than those described above. For example, the terminal 2 may be equipped with multiple cameras and microphones, or may be equipped with an infrared light sensor in addition to a visible light camera.

[0029] FIG. 3 is a diagram illustrating an example of the configuration of the server 3. As shown in FIG.

[0030] The server 3 may be a so-called commercial server, and includes a calculation unit (e.g., a Central Processing Unit) 301, a memory 302, a communication unit 303 for transmitting and receiving data via a network, and a storage unit 304 for storing information, such as a hard disk drive (HDD) or a solid state drive (SSD), all of which are connected by a bus. The server 3 has sufficient resources and capabilities to process a large amount of data input from multiple terminals 2 in parallel.

[0031] The calculation unit 301 implements necessary functions by executing a program stored in the storage unit 304 .

[0032] The memory 302 is a main storage device used when the calculation unit 301 executes processing, and is composed of a volatile storage element such as a RAM (Random Access Memory).

[0033] The communication unit 303 is an interface used by the server 3 to communicate with an external device, and is configured with a NIC (Network Interface Card) etc. The communication unit 303 is connected to a network (e.g., the Internet) and communicates with the external device via the network.

[0034] The storage unit 304 stores processing programs, which are expanded in the memory 302 when executed and operate. The processing programs include programs realizing a face image analysis processing unit 305 that analyzes face movement and blood flow data from image data transmitted from the terminal 2, a voice analysis processing unit 306 that analyzes voice data transmitted from the terminal 2, a transmission / reception processing unit 307 that controls data transmission and reception via the network, the face image analysis processing unit 305, and a behavior analysis processing unit 308 that estimates the behavior of participants from the processing results of the voice analysis processing unit 306.

[0035] The server 3 may further include components other than those described above, such as a display unit and a display processing unit that display information on a display for display.

[0036] In the above configuration, the information processing system 1 can support a conference. The embodiment from the start to the end of the conference is the same as a general remote conference system. The organizer sets the date and time of the conference and the participants, and participants other than the organizer can also access the server 3 from their own terminals 2, allowing all participants to receive support under this system.

[0037] In order to specialize the information processing system 1 for support purposes, it is possible to use it in conjunction with other remote conference systems. Materials can be shared, camera images can be displayed, and voice calls can be made on the remote conference system, and the information processing system 1 can be specialized for calculating and analyzing the status of participants.

[0038] The conference support method in the above configuration will be described with reference to FIG.

[0039] First, a procedure for acquiring information on participants in a conference in the information processing system 1 according to the first embodiment will be described with reference to FIG.

[0040] FIG. 4 is a flowchart illustrating an example of an information acquisition process according to the first embodiment.

[0041] First, the process of steps S401 to S403 and the process of steps S406 to S407 are executed in parallel.

[0042] In step S401, the terminal 2 acquires a facial image of each participant using the imaging unit 203. When the calculation unit 201 instructs the imaging unit 203 to start capturing images, the facial image of each participant is acquired at a predetermined frame rate (e.g., 30 fps), the input processing unit 207 temporarily stores the image in the memory 202, and the transmission / reception processing unit 208 transmits the image stored in the memory 202 to the server 3 via the communication unit 205, thereby performing this process.

[0043] It is not necessary for the transmission / reception processor 208 to transfer all captured images, and it may be possible to thin out and transmit only an amount that can be stably transmitted according to the network bandwidth, or to reduce the size of the image before transmitting it. Also, it is possible to thin out and transmit the image to an extent that does not cause a disadvantage such as a deterioration in the accuracy of gesture detection or pulse rate estimation in the server 3, or to change the size of the image before transmitting it.

[0044] Next, a gesture analysis process, which will be described later with reference to FIG. 5, is executed to analyze the presence or absence of gestures by the participants from the image received by the server 3 from the terminal 2 (step S402).

[0045] Next, a pulse rate estimation process, described later with reference to FIG. 6, is executed to detect facial blood flow information of the participant from the image received by the server 3 from the terminal 2, estimate the pulse rate (step S403), and the process proceeds to step S404.

[0046] In the server 3, the transmission / reception processing unit 307 stores the image received by the communication unit 303 in the memory 302. The face image analysis processing unit 305, for example, constantly monitors whether a new image has been stored in the memory 302, and when it is confirmed that a new image has been added to the memory, executes gesture analysis processing (step S403) and pulse rate estimation processing (step S404). In order to determine whether an image has been added, for example, the transmission / reception processing unit 307 may generate reception time information when receiving an image and record it in the memory 302 together with the image data, and refer to this.

[0047] Meanwhile, in step S406, the terminal 2 acquires voice data of each participant's speech using the microphone 204. When the calculation unit 201 issues an instruction to start inputting to the microphone 204, the input processing unit 207 temporarily stores the voice data in the memory 202, and the transmission / reception processing unit 208 transmits the voice data stored in the memory 202 to the server 3 via the communication unit 205, thereby performing this process.

[0048] Next, the voice data received by the server 3 from the terminal 2 is analyzed by the voice analysis processing unit 306, converted into text information, and stored in the memory 302 or storage unit 304, or in a database contained therein (step S407), and the process proceeds to step S404.

[0049] In step S404, a manner analysis process, which will be described later with reference to FIG. 7, is executed to analyze the time-series data acquired in steps S402, S403, and S407, and estimate the manner of the participants.

[0050] Next, the server 3 notifies the terminal 2 of the estimated state of the participant, updates the display screen of the display unit 210 of the terminal 2, which will be described later with reference to FIG. 9 (step S405), and ends the process.

[0051] In the above explanation, for simplicity, an example of data transmission and reception between one terminal 2 and one server 3 has been described. However, in reality, one server 3 can perform similar processing on N terminals 2 owned by multiple participants taking part in one conference, enabling each participant to grasp the status of all participants taking part in the conference.

[0052] FIG. 5 is a flowchart showing an example of the gesture analysis process in step S402.

[0053] In step S501, the face image analysis processing unit 305 extracts edges (feature points) of each component such as the face outline, eyes, nose, mouth, etc., from the latest image. Methods for extracting face feature points include a method for checking the degree of agreement between a pre-trained model and an object in the image, and publicly known OSS (Open Source Software) may be used.

[0054] Next, the face image analysis processing unit 305 judges whether the extraction of the facial feature points has been successful (step S502). If it is judged in step S502 that the extraction of the facial feature points has been successful, the process proceeds to step S503. On the other hand, if it is not judged in step S502 that the extraction of the facial feature points has been successful, the process ends. If the facial feature points cannot be extracted, it is highly likely that the face is not captured in the image or is not facing the camera, and there is no target for processing.

[0055] In step S503, the face image analysis processing unit 305 stores the coordinate values ​​and time information of the feature points extracted in step S501 in the memory 302 in chronological order.

[0056] Next, the face image analysis processor 305 detects a predetermined gesture of the participant from the coordinate values ​​and time information of each feature point stored in chronological order in the memory 302 (step S504), and proceeds to step S505.

[0057] In step S504, the presence or absence of various gestures is analyzed from the facial images of the participants using recent images from a specified period of time. For example, the coordinate values ​​within a certain period of time, such as from the processing timing to 2 seconds before, may be read, and gestures may be detected from the transition (fluctuation) of the coordinates of each feature point.

[0058] For example, when the coordinates of the facial contour or nose feature point move downward once and then upward, and the change (speed, displacement) is equal to or greater than a certain value, and the change in speed in the horizontal direction is equal to or less than a certain value, it is determined that the participant has made a "nodding" gesture. When the change (speed, displacement) in the left-right direction of the facial contour feature point is equal to or greater than a certain value, and the change in speed in the vertical direction is equal to or less than a certain value, it is determined that the participant has made a "shaking head" gesture. When the difference in the up-down direction between the feature points at the left and right ends of the facial contour increases once to a certain value or more and then returns to the original value (converging to the measurement error range that occurs in a neutral state), it is determined that the participant has made a "tilting head" gesture.

[0059] In addition, if the distance between the feature points at the upper and lower ends of the eyelids falls below a certain value and then returns to normal, the participant is determined to be blinking.

[0060] In addition, if the distance between the feature points of the upper and lower eyelids is equal to or greater than a certain percentage and equal to or less than a certain distance during a specified period of time, it is determined that the eyes are closed. In addition, if the distance between the feature points of the eyebrows and the feature points of the upper eyelids is equal to or greater than a certain distance and equal to or more than a certain percentage of the data to be analyzed, it is determined that the person is "raising their eyebrows." In addition, if the distance between the left and right eyebrows is equal to or greater than a certain percentage and equal to or less than a certain distance during a specified period of time, it is determined that the person is "pushing their eyebrows together."

[0061] In addition, if the face width (the distance between the left and right edges of the face) becomes larger than a certain percentage, it is determined to be "leaning forward." Conversely, if the face width (the distance between the left and right edges of the face) becomes smaller than a certain percentage, it is determined to be "leaning back."

[0062] The threshold value for determining the movement of each facial part may be corrected in proportion to the size of the face image on the screen (or the distance from the camera to the face).

[0063] In addition to the method described above of determining rules for the movement of feature points that indicate gestures, gestures can also be estimated by building an AI model trained using large amounts of video data or time-series data.

[0064] In step S505, the face image analysis processing unit 305 saves the gesture detection result obtained in step S504 in the memory 302 or the storage unit 304, or in a database contained therein, and ends the process. By recording in the storage unit 304 or a database contained therein, it is possible to permanently refer to what gestures the participants made and when even after the conference has ended, which has the effect of enabling analysis of the contents of the conference.

[0065] The information stored in step S505 may include, for example, the name of the participant who performed the gesture, the start time of the gesture, the end time of the gesture, the gesture type, and the number of occurrences. The start time and end time may be recorded as the time elapsed from the start of the conference or as the time on the system clock, and the gesture type may be recorded as a character string or a numerical value with a code associated with each gesture so that it is clear whether a nod, blink, etc., occurred.

[0066] For example, if a nod, a head shake, or the like is recorded every time it occurs, the number of occurrences is set to 1. It is also possible to record how many times a certain gesture occurs within a specified time interval, and to set the start time and end time of an event in which an eyeblink occurs N times (N is a natural number) within a certain period of time (e.g., 30 seconds) to this measurement time interval, and record the number of occurrences as N.

[0067] As described above, the process shown in FIG. 5 makes it possible to detect the gestures of participants by analyzing face images in the server 3. Note that the thresholds for each judgment may be adaptively changed depending on the individual characteristics of the participant's gestures. For example, for a participant who has a habit of nodding with a small movement range, the threshold for nodding detection may be set small. One method for this is to provide a calibration procedure before participating in the conference. For example, participants may be asked to perform gestures such as nodding, shaking their head, and tilting their head in accordance with instructions on the display screen in advance, and the detection threshold may be corrected based on the understanding of each individual's movements.

[0068] FIG. 6 is a flowchart showing an example of the details of the pulse rate estimation process in step S403.

[0069] In step S601, the face image analysis processing unit 305 extracts edges (feature points) of each component such as the face outline, eyes, nose, mouth, etc., from the latest image. Methods for extracting face feature points include a method for checking the degree of agreement between a pre-trained model and an object in the image, and a publicly known OSS may be used.

[0070] Next, the face image analysis processing unit 305 judges whether the extraction of the facial feature points has been successful (step S602). If it is judged in step S502 that the extraction of the facial feature points has been successful, the process proceeds to step S603. On the other hand, if it is not judged in step S502 that the extraction of the facial feature points has been successful, the process ends. If the facial feature points cannot be extracted, it is highly likely that the face is not captured in the image or is not facing the camera, and there is no target for processing.

[0071] In step S603, the face image analysis processing unit 305 detects and analyzes the blood flow distribution of the face part. For detecting the facial blood flow from the video signal, for example, the technology described in JP 2022-119550 A is used.

[0072] According to JP 2022-119550 A, the video signal is separated by color. For example, the blood flow distribution can be obtained using a red signal. When the blood flow distribution is measured in a human body, red light penetrates the human body, some of it is absorbed, and the rest of the light is diffused and reflected. The light emitted from the body can be captured by the visible light imaging unit. When the blood flow changes, the thickness of the blood vessel changes and the amount of blood flowing inside changes, and the optical characteristics change accordingly. This change in optical characteristics also occurs in the red wavelength range that reaches the blood vessels, so the change in blood flow can be seen.

[0073] Next, the face image analysis processing unit 305 estimates the pulse rate from the result obtained in step S603 (step S604).

[0074] According to JP 2022-119550 A, for example, the pulse can be detected by performing a Fourier transform on the change in blood flow over time and performing frequency analysis. In addition, by recording the time when the pulse peaks and recording the change in the pulse period for a sufficient length, the balance of the autonomic nerves can be estimated using the analysis.

[0075] Next, the face image analysis processing unit 305 stores the pulse rate estimated in step S604 in the memory 302 or the storage unit 304, or in a database contained therein (step S605), and ends the process. By recording in the storage unit 304 or in a database contained therein, it is possible to permanently refer to what changes occurred in the participants and when even after the conference has ended, and to perform analysis of the contents of the conference. The contents of the information to be stored include, for example, the name of the participant to be estimated, the measurement start time, the measurement end time, and the estimated pulse rate. The measurement start time and the measurement end time may be recorded as the elapsed time from the start of the conference, or may be recorded as the time of the system clock. The estimated pulse rate may be recorded as the number of fluctuations between the measurement start time and the measurement end time, normalized to the number of fluctuations per minute.

[0076] The face image analysis processing unit 305 repeatedly executes the gesture analysis processing in FIG. 5 and the pulse rate estimation processing in FIG. 6 until the end of the conference.

[0077] In addition to the above, the face image analysis processing unit 305 may also perform functions such as gaze direction estimation and face orientation estimation. As with feature point extraction, it is possible to add these functions by using generally available OSS. For example, as the result of estimating the gaze direction or face orientation, if a participant does not face the image capture unit 203 directly for a certain period of time or if the participant continues to look downward, the participant's name, measurement start time, measurement end time, and number of occurrences are recorded. This enables the conference support system to grasp the participant's condition more accurately.

[0078] FIG. 7 is a flowchart showing an example of the manner analysis process in step S404.

[0079] In step S701, the manner analysis processing unit 308 in the server 3 refers to time information and reads out data to be analyzed from the current time until a certain time ago from a database stored in the memory 302 or the storage unit 304. The data to be analyzed includes gestures, pulse rates, and utterance data converted into text information of all conference participants.

[0080] The behavior analysis process is performed at regular intervals. For example, it may be performed once every three seconds, and the evaluation may be performed by referring to data from the analysis process time up to a certain time before (for example, 30 seconds). This makes it possible to see whether there has been a behavior change in other participants in response to a comment made by a certain participant.

[0081] Although the processing load increases by evaluating the behavioral analysis process at shorter execution intervals, it has the advantage of being able to provide feedback to participants on changes in a short period of time. Since the processing load increases in proportion to the number of participants, by adjusting the execution interval according to the number of participants, it is possible to provide feedback that is appropriate for the participation status.

[0082] Next, the manner analysis processing unit 308 performs a manner analysis using the read data (step S702).

[0083] FIG. 8 is a diagram for explaining the manner analysis in step S702.

[0084] 8 shows an example of the relationship between the detection results of gestures and utterances and the appearance of participants. In the appearance analysis in step S702, for example, if a nodding gesture of a certain participant is detected at a certain analysis timing, it is determined that the participant is positive about the content of the conversation at that time, if a head shaking gesture is detected, it is determined that the participant is negative about the content, and if a head tilting gesture is detected, it is determined that the participant is suspicious.

[0085] The same is true for the results of voice analysis. For example, if the person says "That's right," it is judged to be affirmative; if the person says "That's not it," it is judged to be negative; if the person says "I get it," it is judged to be understanding; and if the person says "Why?" it is judged to be curious.

[0086] The manner may be determined based on such rules, and similar utterances such as "That's right," "No," and "Why not?" may be determined in the same way.

[0087] Moreover, the manner analysis processing unit 308 stores the result of the determination in the memory 302 or the storage unit 304, or in a database contained therein. By recording in the storage unit 304 or in a database contained therein, the manner in which the participants were and when can be permanently referred to even after the end of the meeting, which has the effect of enabling analysis of the efficiency and activity level of the meeting. Note that the efficiency of a meeting can be calculated, for example, based on whether the purpose of the meeting (completion of a progress report or reaching a conclusion that all participants agree on through discussion) was achieved in a short time, and the activity level can be calculated, for example, based on the number of people who spoke during the meeting, the overall amount of speech, and the presence or absence of exciting scenes (such as a certain level of volume or more), but appropriate indicators may be set depending on the agenda and content of the meeting.

[0088] Furthermore, the manner analysis processing unit 308 may determine that if a certain amount of time has passed since a certain participant was once in a certain state (for example, a state of understanding), the state is changed to a neutral state. Alternatively, if a questionable decision occurs once after a state of understanding, the state may be changed to a neutral state. Alternatively, if the agenda is changed, the state of all participants may be changed to neutral. These processes make it possible to more appropriately share each participant's reaction to the meeting.

[0089] Although the rule-based method for determining affirmative / negative and questionable / understood has been described above, the user may have the manner analysis processing unit 308 learn in advance how to associate their own behavior, such as their own utterances and gestures, with the manner. This allows the conference participants to appropriately express their intentions in the situation.

[0090] Returning to FIG. 7, in step S703, the analysis result is notified from the server 3 to the terminal 2, and the detection result is displayed on the screen of each participant, and the process ends.

[0091] In addition, participants who are participating in a conference remotely may use multiple displays in their environment. In this case, the direction of the face or the line of sight may not necessarily be directly facing the imaging unit 203 of the terminal 2, and gestures may not be detected correctly. Therefore, when the face direction or line of sight does not match the camera for a certain period of time in the face image analysis processing unit 305, the state may be communicated to the facilitator so that the facilitator knows that the gesture detection result does not show the actual situation, and information may be presented to encourage the participant to speak periodically regardless of the gesture detection result. This makes it possible to avoid a decrease in efficiency of the conference due to an erroneous determination result.

[0092] FIG. 9 is a diagram showing an example of a display screen shown to the participants in step S405.

[0093] The display screen 9 is provided with several display areas, and the display contents are divided according to function. In Fig. 9, as an example, a mode map display area 901, a conference progress display area 902, and a remarks record display area 903 are provided.

[0094] The modality map display area 901 displays the modality of each participant on a map based on the processing results by the modality analysis processing unit 308. The modality map display area 901 is divided into nine areas, for example, with three vertical rows representing modality classifications such as positive reaction to the ongoing topic, no reaction (neutral), and negative reaction. Also, three horizontal columns representing modality classifications such as reaction indicating doubt about the ongoing topic, no reaction (neutral), and reaction indicating understanding.

[0095] Whether each participant is in a positive or negative state regarding the agenda, or whether they understand or have doubts about the agenda, is shown by displaying an icon of each participant in the appropriate one of the nine areas.

[0096] At the start of the meeting, all the icons of the participants are displayed in a neutral area, and for example, if the result of the manner analysis process in step S404 indicates that a certain participant understands the ongoing discussion, the display position of that participant's icon is updated to within the same row, one column to the right. Alternatively, if it is determined that a certain participant has responded positively to the ongoing discussion, the display position of that participant's icon is updated to the row above in the same row. If a participant's icon is already displayed in the top row and there is a positive response to the agenda, the display position of the icon is not changed.

[0097] Furthermore, if the result of the determination in the manner analysis process step S404 is that no manner of any of the participants is detected for a predetermined period of time, the display position of the icon may be updated to a row or column indicating neutral.

[0098] By visualizing the state of each participant on a state map as in the above example, meeting participants, especially the facilitator, can check the state of each participant even when they are not face to face. Furthermore, the facilitator can ask participants who show doubts or negative reactions to speak up, enabling more accurate running of the meeting.

[0099] The progress of the conference is displayed in the conference progress display area 902. The conference progress display area 902 displays the schedule based on a schedule prepared in advance by the organizer, and the facilitator can check a topic every time the topic is finished, thereby making the progress visible to all participants.

[0100] The process of reflecting the facilitator's operations on the screen can be achieved using a general server-client communication function, and the server 3 can issue a screen update command to each terminal 2 used by all participants in accordance with the operations.

[0101] The speech record display area 903 visualizes the comments of each participant by displaying text information converted from voice data by the voice analysis processor 306. The transmission / reception processor 208 of the terminal 2 stores the text information received from the server 3 in the memory 202, and if there is an update to the text information in the memory 202, the display processor 209 outputs it together with the names of the participants in the speech record display area 903, which is displayed on the display screen 9. This has the effect of allowing participants to look back and check the content of utterances made during the conference, and by extracting important points from the speech record after the conference has ended, it is possible to easily create minutes, which has the effect of improving work efficiency.

[0102] Furthermore, when a participant places the mouse cursor over a participant's icon on the display unit 210 of his / her own terminal 2, an icon 904 that matches the state determination result shown in FIG. 8 is displayed.

[0103] In FIG. 8, the icons in the first row indicate a neutral state of the participant and are displayed when none of the following applies. The icons in the second row indicate a positive state of the participant and are displayed when, for example, a nodding gesture is recognized. The icons in the third row indicate a negative state of the participant and are displayed when, for example, a head shaking gesture is recognized. The icons in the fourth row indicate a state of understanding of the participant and are displayed when, for example, an utterance of "I understand." The icons in the fifth row indicate a state of questioning of the participant and are displayed when, for example, a head shaking gesture is recognized.

[0104] In the example of Figure 9, the reason why the display position of the participant's icon moved to the negative side is that a head-shaking gesture was detected. When the participant's state changes due to their comment, the content of their comment may be displayed in a speech bubble. This makes it possible to visualize the reason for the state change in an easy-to-understand way.

[0105] According to the process described in FIG. 7 to FIG. 9, the state of a participant is estimated based on a predetermined gesture or a predetermined statement, and the estimated state of the participant is displayed on the display unit 210, thereby improving the liveliness of the conference.

[0106] Although the above describes an example of estimating a participant's condition based on a single detection result of a keyword from gesture analysis and text information converted from voice data, it is also possible to estimate a participant's condition based on a combination of elements.

[0107] For example, it is known that facial blood flow is correlated with pulse rate, and that a decrease in pulse rate is correlated with a decrease in interest and attention. Blinks are also related to cognitive processes, and it is also known that a decrease in the blink rate (the number of blinks per unit time) is correlated with an increase in drowsiness and fatigue. Therefore, by combining the information on the fluctuation in pulse rate estimated from facial blood flow and the fluctuation in the blink rate detected from the camera image, it may be possible to estimate the interest in the topic during the meeting and the presence or absence of drowsiness. With reference to FIG. 10, a description will be given of a state determination process that combines the fluctuation in pulse rate and the fluctuation in the blink rate.

[0108] Fig. 10 is a flowchart showing an example of a mode determination process. Note that the mode determination process of Fig. 10 may be performed simultaneously with the mode analysis process of Fig. 7, or these processes may be performed sequentially.

[0109] In step S1001, similarly to step S701, the manner analysis processing unit 308 in the server 3 refers to time information and reads out data to be analyzed from the current time up to a certain time ago from the database stored in the memory 302 or the storage unit 304.

[0110] Next, similarly to step S702, the manner analysis processing unit 308 performs a manner analysis using the read data (step S1002).

[0111] In step S1003, it is determined whether or not the result of the morphology analysis in step S702 shows that both the estimated pulse rate and the blink rate have increased by a predetermined threshold or more, compared with the result of the previous morphology analysis.

[0112] For example, the pulse rate and blink rate for the previous minute are calculated every 10 seconds, and if there is a 20% or more increase compared to the previous estimated pulse rate and blink rate, it is determined that the estimated pulse rate and blink rate have increased by a predetermined threshold or more. An increase in pulse rate and blink rate may indicate an increase in mental tension, and may have caused some emotion due to the content of the discussion at that time in a meeting.

[0113] If it is determined in step S1003 that both the estimated pulse rate and the blink rate have increased by a predetermined threshold or more compared to the results of the previous morphology analysis, the process proceeds to step S1004. On the other hand, if it is not determined in step S1003 that both the estimated pulse rate and the blink rate have increased by a predetermined threshold or more compared to the results of the previous morphology analysis, the process ends.

[0114] In step S1004, the manner analysis processing unit 308 reflects this change in manner on the display screen of the display unit 210 of the terminal 2, which will be described later with reference to FIG. 11, and ends the process.

[0115] 10, an example is described in which it is determined that a participant is interested in the agenda when both the estimated pulse rate and the blink rate increase by a predetermined threshold or more compared to the results of the previous behavioral analysis, but it is also possible to perform a process in which it is determined that a participant is sleepy and cannot concentrate on the meeting and display this on the screen when both the estimated pulse rate and the blink rate decrease by a predetermined threshold or more compared to the results of the previous behavioral analysis. A tendency for the pulse rate to decrease suggests that the participant may be in a resting state, and a tendency for the blink rate to decrease suggests that the participant may be in a resting state with reduced mental tension due to sleepiness.

[0116] Also, when it is detected that the participant has closed his / her eyes for a certain period of time, it may be determined that the participant is not concentrating on the conference, and a process of displaying this on the screen may be performed.

[0117] In addition, if the pulse rate and the blink rate continue to exceed a predetermined number, it may be determined that the participant is in a state of mental tension. If the participant is in a state of mental tension, there is a concern that the participant may not be able to perform well in presentations or discussions during the meeting.

[0118] Fig. 11 is a diagram showing an example of a display screen shown to the participants in step S1004. Based on the display in Fig. 9, the display example of the detection result explained in Fig. 10 is additionally shown. In Fig. 11, the same components as those in Fig. 9 are given the same reference numerals, and the explanation thereof will be omitted.

[0119] Icon 1101 indicates that the participant is interested in the ongoing topic and intends to speak. In the state determination process of Fig. 10, when the pulse rate and blink rate increase by more than a threshold value, a change in the emotion of the participant is detected, and icon 1101 is displayed. This allows the facilitator to use it as an aid in deciding whether to encourage the participant to speak.

[0120] Icon 1102 indicates that the participant is feeling drowsy and is unable to concentrate. In the state determination process of FIG. 10, when the pulse rate and blink rate decrease by more than a threshold value, and a change in the participant's decreased concentration is detected, icon 1102 is displayed. This can be used to help the facilitator decide whether to encourage the participant to speak, or the participant himself can correctly recognize his own state. Alternatively, terminal 2 can alert the participant with an alert sound or the like.

[0121] When the facial blood flow and blink rate of the participants are continuously measured and the rate of change is calculated, the interest in the topic and the level of drowsiness based on the rate of change may be classified (for example, "high," "normal," "low," etc.) and the degree may be displayed on the display unit 210. This allows the facilitator to provide more suitable support, for example, by adjusting the progress of the meeting or the selection of topics based on the information obtained from the display unit 210, or by alerting the participant with an alarm sound when the participant's drowsiness level is high.

[0122] Also, if a participant continues to be in a tense state with high pulse rate and / or blink rate for a certain period of time, a message encouraging the participant to take a deep breath may be displayed on the participant's screen. The display method may be within the screen shown in FIG. 11 or a new pop-up screen. This allows for more suitable support, such as relieving the participant of tension.

[0123] Icon 1103 indicates that the participant is not facing the screen because he / she is looking at another display, etc. Icon 1103 is displayed when facial feature points cannot be extracted by the gesture analysis process in Fig. 5 and the pulse rate estimation process in Fig. 6. This allows the facilitator to notice that the gestures and pulse rate of the participant cannot be detected, and to use this as an aid in making decisions such as encouraging the participant to speak at appropriate intervals.

[0124] By displaying these icons, various states of the participants can be understood, and the conference can be run more smoothly. Note that the display of these icons may be limited to only some of the participants (for example, only the facilitator).

[0125] According to the process of FIGS. 10 to 11, the state of the participant is estimated based on the number of blinks and the estimated pulse rate, and the estimated state of the participant is displayed on the display unit 210, thereby making it possible to improve the liveliness of the conference.

[0126] In the above embodiment, an embodiment has been described in which an image is transmitted from the terminal 2 to the server 3 and the image is analyzed by the server 3, but the same processing is possible by performing face image analysis processing in the terminal 2 and transmitting calculated feature amount data to the server 3. Similarly, processing is also possible by performing voice analysis processing in the terminal 2 and transmitting transcribed data to the server 3. EXAMPLES

[0127] In the first embodiment, the server 3 executes all server-side processing. However, in consideration of the increased processing load of the software, it is possible to distribute the functions to a plurality of servers.

[0128] FIG. 12 is a block diagram illustrating an example of a configuration of an information processing system 1 according to the second embodiment.

[0129] The information processing system 1 has a video analysis processing server 5 whose function is limited to video analysis processing, a voice analysis processing server 6 whose function is limited to voice analysis processing, and a modal analysis processing server 7 whose function is limited to modal analysis processing, and by transmitting and receiving the necessary information between each server, it is possible to realize functions equivalent to those of Example 1.

[0130] That is, image data acquired by the input process 21 in the terminal 2 is transmitted to the video analysis processing server 5 via the communication unit 205 by the transmission / reception process 22 .

[0131] In the video analysis processing server 5 , the video analysis processing 51 executes gesture analysis and pulse rate estimation processing in the same manner as in the first embodiment, and the transmission / reception processing 52 transmits the feature amount of the analysis result to the manner analysis processing server 7 .

[0132] The mode analysis processing server 7 executes a mode analysis process 71, and transmits the mode analysis result to the terminal 2 by a transmission / reception process 72. In the terminal 2, the result can be displayed on the display unit 210 by a display process 23.

[0133] Furthermore, the voice data acquired by the input process 21 in the terminal 2 is transmitted to the voice analysis processing server 6 via the communication unit 205 by the transmission / reception process 22. The voice analysis processing server 6 executes the voice analysis processing unit 61, and transmits the analysis results and text information to the manner analysis processing server 7 by the transmission / reception process 62.

[0134] According to this embodiment, the functions of video analysis processing, audio analysis processing, and modality analysis processing are distributed among a plurality of servers, so that the processing load of the software on each server can be reduced.

[0135] The present invention is not limited to the above-described embodiments, and includes various modified examples. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the configurations described. It is also possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace a part of the configuration of each embodiment with another configuration. [Explanation of symbols]

[0136] 1...information processing system, 2...terminal, 201...computing unit, 202...memory, 203...imaging unit, 204...microphone, 205...communication unit, 206...storage unit, 207...input processing unit, 208...transmission and reception processing unit, 209...display processing unit, 210...display unit, 3...server, 301...computing unit, 302...memory, 303...communication unit, 304...storage unit, 305...face image analysis processing unit, 306...audio analysis processing unit, 307...transmission and reception processing unit, 308...manner analysis processing unit, 4...network, 5...video analysis processing server, 6...audio analysis processing server, 7...manner analysis processing server

Claims

1. An imaging unit for acquiring face images of conference participants; a face image analysis processing unit for acquiring gesture features and pulse rates of the conference participants from the face images; a behavior analysis processing unit that estimates a behavior of the conference participant based on the gesture feature amount and the pulse rate; A display unit that displays the estimated state of the conference participant.

2. 2. The information processing system according to claim 1, An information processing system in which the feature of the gesture is the number of blinks.

3. 3. The information processing system according to claim 2, The information processing system infers that the conference participants are interested in the topic when the number of blinks increases by more than a predetermined threshold value at a predetermined time interval and the pulse rate increases by more than a predetermined threshold value.

4. 3. The information processing system according to claim 2, The information processing system wherein the behavior analysis processing unit estimates that the conference participant is in a drowsy state when the number of blinks decreases by more than a predetermined threshold and the pulse rate decreases by more than a predetermined threshold at a predetermined time interval.

5. 3. The information processing system according to claim 2, The manner of analysis processing unit estimates that the conference participant is in a continuing state of mental tension if the pulse rate and the number of blinks remain equal to or greater than a predetermined number.

6. 2. The information processing system according to claim 1, The facial image analysis processing unit detects a predetermined gesture of the conference participant from the facial image, The manner analysis processing unit is an information processing system that estimates the manner of the conference participant based on the detected predetermined gesture.

7. 2. The information processing system according to claim 1, A voice acquisition unit that acquires voice data; A voice analysis processing unit that detects a predetermined utterance from the voice data, The behavior analysis processing unit is an information processing system that estimates the behavior of the conference participant based on the detected predetermined utterance.

8. An imaging step of acquiring face images of conference participants; a face image analysis process step of acquiring gesture features and pulse rates of the conference participants from the face images; a behavior analysis processing step of estimating a behavior of the conference participant based on the gesture feature amount and the pulse rate; and a display step of displaying the estimated state of the conference participant.

9. 9. The information processing method according to claim 8, The information processing method, wherein the feature of the gesture is the number of blinks.

10. 10. The information processing method according to claim 9, In the manner analysis processing step, when the number of blinks increases by more than a predetermined threshold within a predetermined time interval and the pulse rate increases by more than a predetermined threshold, the information processing method estimates that the conference participant is interested in the topic.

11. 10. The information processing method according to claim 9, In the manner analysis processing step, when the number of blinks decreases by more than a predetermined threshold within a predetermined time interval and the pulse rate decreases by more than a predetermined threshold, the information processing method estimates that the conference participant is in a drowsy state.

12. 10. The information processing method according to claim 9, In the manner analysis processing step, if the pulse rate and the number of blinks continue to exceed a predetermined number, it is estimated that the mental tension of the conference participant continues.

13. 9. The information processing method according to claim 8, In the face image analysis processing step, a predetermined gesture of the conference participant is detected from the face image; In the manner analysis processing step, a manner of the conference participant is estimated based on the detected predetermined gesture.

14. 9. The information processing method according to claim 8, A voice acquisition step of acquiring voice data; A voice analysis processing step of detecting a predetermined utterance from the voice data, In the manner analysis processing step, a manner of the conference participant is estimated based on the detected predetermined utterance.

Images