Automatic meeting minutes creation system, automatic meeting minutes creation method, and program

Abstract

We provide an automated meeting minutes creation system that can include comments made by attendees who were not registered in advance during a meeting. [Solution] The automated meeting minutes creation system of this disclosure comprises: a registration unit for registering one or more attendees attending a meeting in advance; an acquisition unit for acquiring video and audio data during the meeting; a facial recognition unit for determining speakers by facial recognition based on the video data; a meeting minutes creation unit for creating meeting minutes based on the determined speakers and the content of their statements; an identification unit for identifying the speaker as an unregistered attendee if the determined speaker is not one of the one or more attendees; a storage unit for storing the content of the unregistered attendee's statements based on the unregistered attendee's face and clothing; and a determination unit for determining who the unregistered attendee is based on the video and audio data during the meeting. The meeting minutes creation unit updates the meeting minutes based on the determination result of the determination unit.

Description

Technical Field

[0001] The present disclosure relates to a minutes automatic creation system, a minutes automatic creation method, and a program.

Background Art

[0002] In Patent Document 1, in an online video conference, an image related to each attendee is recognized from video data, voice data of each attendee is acquired, and the voice data is compared with the feature information of the voice of each attendee registered in advance to identify the speaker of each utterance in the voice data. A system is disclosed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in an online meeting as described above, there may be attendees who are not registered in advance, and there is a problem that minutes including such unregistered attendees cannot be created. The technique described in Patent Document 1 does not assume such a problem, and no solution to this problem is disclosed.

[0005] The present disclosure has been made to solve such problems, and an object thereof is to provide a minutes automatic creation system, a minutes automatic creation method, and a program capable of reflecting the utterances of attendees who are not registered in advance in the minutes of a meeting.

Means for Solving the Problems

[0006] The minutes automatic creation system according to one aspect of the present disclosure is A registration section for registering one or more attendees who will be attending the meeting, An acquisition unit that acquires video and audio data during a meeting, A facial recognition unit that determines the speaker based on video data, Based on the decided speakers and the content of their statements, the minutes-taking department will prepare the minutes. If the person who was selected to speak is not one of the attendees, the identification unit will identify that person as an unregistered attendee. A memory unit that stores the content of statements made by identified unregistered attendees based on their faces and clothing, A determination unit that determines who the identified unregistered attendees are based on video and audio data from the meeting, Equipped with, The minutes preparation department updates the minutes based on the judgment results of the evaluation department.

[0007] One aspect of this disclosure describes an automated method for creating meeting minutes, Steps to acquire video and audio data during the meeting, The steps include determining the speaker through facial recognition based on video data, The steps include creating meeting minutes based on the decided speakers and the content of their statements, If the person who was selected to speak is not one of the attendees, the next step is to identify that person as an unregistered attendee. Based on the face and clothing of the identified unregistered attendee, the process involves remembering the content of what the identified unregistered attendee said, A step to determine who the identified unregistered attendees are based on video and audio data from the meeting, The steps include updating the meeting minutes based on the judgment result of the judgment unit, It includes.

[0008] A program in one aspect of this disclosure is On the computer, The process of acquiring video and audio data during a meeting, A process that determines the speaker using facial recognition based on video data, The process of creating meeting minutes based on the determined speakers and the content of their statements, If the person who was selected to speak is not one of the attendees, the process will be to identify that person as an unregistered attendee. Based on the face and clothing of the identified unregistered attendee, the process involves recording the content of what the identified unregistered attendee said. A process to determine who the identified unregistered attendees are based on video and audio data from the meeting, Based on the judgment result of the judgment unit, the process of updating the minutes, This is what causes it to execute. [Effects of the Invention]

[0009] According to this disclosure, it is possible to provide an automated meeting minutes creation system, a method for automated meeting minutes creation, and a program that can reflect in the meeting minutes statements of attendees who were not registered in advance. [Brief explanation of the drawing]

[0010] [Figure 1] This block diagram shows an example of the configuration of the automated meeting minutes creation system related to this disclosure. [Figure 2] Figure 1 is a flowchart showing an example of the overall process performed by the automated meeting minutes creation system. [Figure 3] Figure 1 is a flowchart illustrating an example of the meeting minutes creation process performed by the automated meeting minutes creation system 1. [Figure 4] Figure 1 is a flowchart illustrating an example of the process for determining unregistered attendees performed by the automated meeting minutes creation system 1. [Figure 5] This block diagram shows an example of the configuration of the automated meeting minutes creation system related to this disclosure. [Figure 6] Figure 5 is a flowchart illustrating an example of the meeting minutes creation process performed by the automated meeting minutes creation system. [Figure 7]It is a flowchart showing an example of the emotion change display process executed by the minutes automatic creation system shown in FIG. 5. [Figure 8] It is a block diagram showing an example of the hardware configuration of the minutes automatic creation system of the present disclosure.

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. However, the invention according to the claims is not limited to the following embodiments. Also, not all of the configurations described in this embodiment are necessarily essential as means for solving the problems. For the sake of clarity of explanation, the following description and drawings have been appropriately omitted and simplified. In each drawing, the same reference numerals are assigned to the same elements, and duplicate explanations are omitted as necessary.

[0012] Each drawing referred to in the embodiments is merely an example for explaining one or more embodiments. Each drawing is not associated with only one specific embodiment, but may be associated with one or more other embodiments. As can be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with the features or steps shown in one or more other drawings, for example, to create embodiments that are not explicitly illustrated or described. Not all of the features or steps shown in any one drawing for explaining exemplary embodiments are necessarily essential, and some features or steps may be omitted. The order of the steps described in any drawing may be changed as appropriate.

[0013] <Background Leading to the Present Disclosure> In recent years, while online meetings are frequently held, there is a problem that there are many meeting memos in which it is not known who among the multiple attendees spoke. Therefore, there is also a system that allows each attendee who participates in an online meeting to register in advance, for example, their face, voice, name, etc.

[0014] On the other hand, some attendees may be hesitant to register their face or voice in advance. Furthermore, it is possible that such unregistered attendees may speak during the meeting. Considering these circumstances, the inventors have concluded that a system that can create meeting minutes including the statements of unregistered attendees and add the names (full names) and job titles of unregistered attendees during or after the meeting, as needed, would be useful.

[0015] Furthermore, because the emotions of the speakers during the meeting cannot be reflected in the minutes, the nuances of what was said are not accurately conveyed, and the resulting minutes cannot adequately provide the information discussed in the meeting.

[0016] To address this issue, the inventors have come to believe that it is also useful to predict the speaker's emotions based on their attitude, facial expressions, reactions, or gestures during their speech, as well as their tone of voice, and to reflect these in the meeting minutes.

[0017] This disclosure provides an automated meeting minutes generation system that can create useful minutes even when there are unregistered attendees in online meetings, and that can incorporate the emotions of the speakers into those minutes. While this disclosure uses a meeting or online meeting as an example, the technology described is applicable not only to meetings but also to online classes in schools, online learning such as certification courses, and more.

[0018] (Embodiment 1) <Configuration of the automated meeting minutes creation system> First, the configuration of the automated meeting minutes creation system according to this embodiment will be described. Figure 1 is a block diagram showing an example of the configuration of the automated meeting minutes creation system according to this disclosure. As shown in Figure 1, the automated meeting minutes creation system 1 comprises a registration unit 10, an acquisition unit 20, a facial recognition unit 30, a meeting minutes creation unit 40, a identification unit 50, a storage unit 60, and a determination unit 70.

[0019] The registration unit 10 is configured to register one or more attendees in advance. Specifically, the registration unit 10 registers the face and clothing of each attendee of the online meeting in the automatic meeting minutes creation system 1 via the camera function of their personal computer, mobile device, or other terminal device (not shown). The registration unit 10 may also register the voice of the attendee in the automatic meeting minutes creation system 1 via the microphone function of their terminal device.

[0020] The acquisition unit 20 is configured to acquire video and audio data during the meeting via each terminal device once the meeting begins. The acquired video and audio data is stored in the storage unit 60.

[0021] The facial recognition unit 30 is configured to determine the speaker by facial recognition based on video data when an attendee speaks using any of the terminal devices. For example, the facial recognition unit 30 performs facial recognition by comparing the facial data of each attendee, which has been pre-registered by the registration unit 10, with the facial image captured by the camera of the terminal device into which the audio data is input.

[0022] The minutes creation unit 40 is configured to create minutes based on the speakers and the content of their statements determined by the facial recognition unit 30. The minutes data includes the speakers' names and job titles, as well as text data of their statements.

[0023] The identification unit 50 is configured to identify a speaker as an unregistered attendee if the speaker determined by facial recognition by the facial recognition unit 30 is not one of the attendees previously registered by the registration unit 10. In the automatic minutes creation system 1 according to this embodiment, the accuracy of the minutes is improved by identifying unregistered attendees, as will be described later.

[0024] As described above, the storage unit 60 is configured to store the video data and audio data acquired by the acquisition unit 20. The video data and audio data thus stored are output to the minutes creation unit 40. Based on this data, the minutes creation unit 40 links the speakers with the text of their statements and creates the minutes.

[0025] Furthermore, the storage unit 60 acquires and stores data of the face and clothing of unregistered attendees based on video data acquired by the acquisition unit 20 via the terminal device of the unregistered attendee identified by the identification unit 50. The storage unit 60 is configured to store the content of the statements made by the identified unregistered attendees in association with the face and clothing data of the identified unregistered attendees. The minutes creation unit 40 also creates minutes for the statements made by the identified unregistered attendees, linking the speaker with the text of their statements.

[0026] The determination unit 70 is configured to determine who the unregistered attendees identified by the identification unit 50 are, based on the video and audio data from the meeting. For example, the determination unit 70 can obtain names and job titles that have not been previously registered in the registration unit 10 from the audio data from the meeting, and then determine who the unregistered attendees are based on the obtained names and job titles.

[0027] Furthermore, the determination unit 70 may determine who the unregistered attendee is based on the content of statements made by other attendees before and after the statement made by the unregistered attendee identified by the identification unit 50. For example, if an unregistered attendee speaks after an attendee has said, "What do you think, Mr. / Ms. XX?", the determination unit 70 can determine that the unregistered attendee is Mr. / Ms. XX. Also, for example, if an unregistered attendee speaks and then another attendee says, "Do you have any comments on Mr. / Ms. XX's opinion?", the determination unit 70 can determine that the unregistered attendee is Mr. / Ms. XX.

[0028] Furthermore, the automated meeting minutes creation system 1 may, depending on the determination result of the determination unit 70, display an image of the face of an unregistered attendee along with a text comment such as "Is this person Mr. / Ms. XX?" on the display screen of the terminal device of all attendees at the meeting. The responses from other attendees to this display can help to more accurately identify unregistered attendees.

[0029] <How the automated meeting minutes creation system works> Next, the operation of the automated meeting minutes creation system 1 according to this embodiment will be described. Figure 2 is a flowchart showing an example of the overall processing performed by the automated meeting minutes creation system 1 shown in Figure 1.

[0030] In the overall process, first, the registration unit 10 registers one or more attendees who will be attending the meeting (step S11). Then, the automated meeting minutes creation system 1 determines whether or not the meeting has started (step S12).

[0031] After registering attendees, the system waits until the meeting begins in step S12. In step S12, if it determines that the meeting has started, the automated minutes creation system 1 executes the minutes creation process (step S13). In this minutes creation process, as will be described later, minutes are created, but it is sufficient that the minutes are updated each time an attendee speaks.

[0032] Next, the automated meeting minutes creation system 1 determines whether the meeting has ended (step S14). If it determines that the meeting has not ended, the processing flow proceeds to step S13, and the meeting minutes creation process is repeatedly executed each time an attendee speaks.

[0033] On the other hand, if it is determined that the meeting has ended, the processing flow moves to step S15, where the storage unit 60 saves the meeting minutes data created by the meeting minutes creation process (step S15), and the automatic meeting minutes creation system 1 terminates this entire process.

[0034] Next, we will explain the meeting minutes creation process. Figure 3 is a flowchart showing an example of the meeting minutes creation process performed by the automated meeting minutes creation system 1 shown in Figure 1. The meeting minutes creation process is performed each time an attendee makes a statement during the meeting.

[0035] In the meeting minutes creation process, first, the acquisition unit 20 acquires video and audio data from the meeting (step S21). The automatic meeting minutes creation system 1 determines, based on the acquired audio data, whether or not any attendees made a statement (step S22). If it is determined that no one made a statement, the processing flow returns to step S21, and steps S21 and S22 are repeated until a statement is made.

[0036] On the other hand, if it is determined that any attendee has made a statement, the processing flow moves to step S23, where the facial recognition unit 30 determines the speaker by facial recognition based on the video data (step S23). Then, the minutes creation unit 40 creates the minutes based on the speaker and the content of their statement (step S24). Here, the minutes creation unit 40 sequentially adds the speakers and the content of their statements during the meeting to the minutes based on the video and audio data acquired by the acquisition unit 20. Then, the automatic minutes creation system 1 terminates this minutes creation process. As described above, the automatic minutes creation system 1 repeatedly executes this minutes creation process until it determines that the meeting has ended.

[0037] Next, we will explain the process for determining unregistered attendees. Figure 4 is a flowchart showing an example of the process for determining unregistered attendees performed by the automated meeting minutes creation system 1 shown in Figure 1. The process for determining unregistered attendees is performed each time any attendee speaks at the meeting. That is, the process for determining unregistered attendees is started in step S22 of the meeting minutes creation process shown in Figure 3 when it is determined that any attendee has spoken.

[0038] In the process of determining whether an attendee is not registered, first, the identification unit 50 determines whether the speaker is an attendee who has been registered in advance, based on the authentication result of the facial recognition unit 30 (step S31). If it is determined that the speaker is an attendee who has been registered in advance, the automatic minutes creation system 1 terminates this process of determining whether an attendee is not registered.

[0039] On the other hand, if the system determines that the speaker is not a pre-registered attendee, the identification unit 50 identifies the speaker as an unregistered attendee (step S32). The acquisition unit 20 acquires data on the unregistered attendee's face and clothing based on the video data (step S33), and the storage unit 60 stores the content of the unregistered attendee's statement in association with the acquired face and clothing data (step S34).

[0040] Next, the determination unit 70 determines who the unregistered attendee is based on the video and audio data from the meeting (step S35). Specifically, the determination unit 70 determines who the unregistered attendee is based on the name or job title acquired by the acquisition unit 20 and the content of other attendees' statements before and after the unregistered attendee's statements. If the determination step is not possible during the meeting, the determination unit 70 may output the determination result to the minutes creation unit 40 as, for example, unregistered attendee "A". "A" indicates information that identifies the unregistered attendee. In that case, the determination unit 70 will continue to determine that the same unregistered attendee's statements are unregistered attendee "A" until the unregistered attendee is determined (identified). The determination unit 70 outputs the determination result to the minutes creation unit 40.

[0041] Next, the minutes creation unit 40 updates the minutes based on the determination result of the determination unit 70 so that the unregistered attendee becomes the determined attendee (step S36). Then, the automatic minutes creation system 1 terminates this unregistered attendee determination process. Note that the unregistered attendee determination process is executed repeatedly until the meeting ends. If the meeting ends while the minutes creation unit 40 is unable to determine who unregistered attendee "A" is, it creates minutes that include unregistered attendee "A". If the minutes creation unit 40 has created minutes that include unregistered attendee "A", it may accept input for the name of unregistered attendee "A". If the name of unregistered attendee "A" is entered, the minutes creation unit 40 may update unregistered attendee "A" with the entered name. For example, a user of the minutes creation system 1 may select unregistered attendees to be included in the created minutes and enter the names of the selected unregistered attendees.

[0042] As described above, the automatic meeting minutes creation system 1 according to this embodiment is configured to include a registration unit 10 for registering one or more attendees in advance, an acquisition unit 20 for acquiring video and audio data during the meeting, a facial recognition unit 30 for determining speakers by facial recognition based on the video data, a meeting minutes creation unit 40 for creating meeting minutes based on the determined speakers and the content of their statements, an identification unit 50 for identifying the speaker as an unregistered attendee if the determined speaker is not one of the one or more attendees, a storage unit 60 for storing the content of the identified unregistered attendee's statements based on the face and clothing of the identified unregistered attendee, and a determination unit 70 for determining who the identified unregistered attendee is based on the video and audio data during the meeting. The meeting minutes creation unit 40 is configured to update the meeting minutes based on the determination result of the determination unit 70. By configuring the automatic meeting minutes creation system 1 according to this embodiment in this way, the content of statements made by attendees who are not registered in advance and the video data at that time can be stored in real time. This allows the system to collectively store the statements of unregistered attendees based on information such as their faces, clothing, and voice tone. Thus, according to the automatic meeting minutes creation system 1 of this embodiment, during or after the meeting, the determination unit 70 determines whether to link the statements of unregistered attendees with their names, job titles, etc., thereby completing the meeting minutes. The complete meeting minutes can then be shared among all relevant parties, including the meeting attendees.

[0043] Furthermore, the automated meeting minutes creation system 1 according to this embodiment can achieve the same effect even when multiple attendees are gathered in a conference room to participate in the meeting, in addition to those attending online. In this case, for example, all attendees in the conference room can be filmed with a video camera, and microphones can be placed next to each attendee.

[0044] Furthermore, the meeting minutes automatic creation system 1 according to this embodiment can be applied not only to so-called meetings, but also to online classes where students attend both at home and in the classroom, and to online learning where students attend both at home and in the classroom (for example, classes at cram schools and various vocational schools).

[0045] (Embodiment 2) <Configuration of the automated meeting minutes creation system> The configuration of the automated meeting minutes creation system according to Embodiment 2 will now be described. Figure 5 is a block diagram showing an example of the configuration of the automated meeting minutes creation system according to this disclosure. As shown in Figure 5, the automated meeting minutes creation system 2 according to this embodiment includes a registration unit 10, an acquisition unit 20, a facial recognition unit 30, a meeting minutes creation unit 40, an identification unit 50, a storage unit 60, a determination unit 70, a video analysis unit 80, and an emotion prediction unit 90. Note that the configurations of the registration unit 10, acquisition unit 20, facial recognition unit 30, meeting minutes creation unit 40, identification unit 50, storage unit 60, and determination unit 70 in the automated meeting minutes creation system 2 have the same functions as the corresponding components of the automated meeting minutes creation system 1 of Embodiment 1, so detailed explanations of them will be omitted as appropriate, except for functions specific to this embodiment.

[0046] The automated meeting minutes creation system 2, like the automated meeting minutes creation system 1, uses a determination unit 70 to determine unregistered attendees who spoke during the meeting, and the meeting minutes creation unit 40 updates the minutes based on the determination result. In addition, the automated meeting minutes creation system 2 is characterized by predicting the emotions of the speakers and reflecting them in the minutes. In this embodiment, the latter will be described in particular in detail.

[0047] The video analysis unit 80 is configured to extract at least one of the speaker's attitude, facial expression, or reaction during the speech, as determined by the facial recognition unit 30, based on the video data acquired by the acquisition unit 20. The video analysis unit 80 may also be configured to further extract the speaker's gestures in the video during the speech, as determined by the facial recognition unit 30, based on the video data acquired by the acquisition unit 20.

[0048] The emotion prediction unit 90 is configured to predict the speaker's emotions based on at least one of the speaker's attitude, facial expression, or reaction extracted by the video analysis unit 80, and a voice analysis of the speaker's remarks. The voice analysis can be any analysis of the speaker's voice tone (including voice color, intonation, tone of voice, and argumentation).

[0049] For example, the emotion prediction unit 90 may predict the speaker's emotions (emotional state) by analyzing the speaker's facial expressions and voice tone during their speech. Alternatively, the emotion prediction unit 90 may further consider the speaker's gestures extracted by the video analysis unit 80 to predict the speaker's emotions. By further analyzing and incorporating the speaker's gestures in this way, the accuracy of predicting the speaker's emotions can be improved, enabling more accurate emotional predictions.

[0050] In this embodiment, the minutes creation unit 40 is configured to create minutes by adding information about the speaker's emotions, as predicted by the emotion prediction unit 90, along with the speaker and the content of their remarks. In this way, the minutes creation unit 40 adds the speaker's emotions to the minutes in addition to the speaker and the content of their remarks.

[0051] The display unit 100 is configured to display the minutes created by the minutes creation unit 40 during the meeting in real time, including the predicted emotions of the speakers. The minutes displayed by the display unit 100 are also displayed in real time on the terminal devices of the meeting attendees. This allows other attendees to grasp the emotional state of the speakers during the meeting in real time.

[0052] Here, the emotion prediction unit 90 may be configured to determine whether the predicted emotion of the speaker has changed since the previous time. Specifically, the emotion prediction unit 90 extracts the previously predicted emotion of the speaker from the minutes created by the minutes creation unit 40 and determines whether the emotion of the currently predicted speaker has changed from the previously predicted emotion of the speaker. Note that this change in emotion includes not only the change in emotion between the previous and current statements by the same speaker, but also cases where the previously predicted emotion for a statement by one speaker has changed from the predicted emotion for a statement by another speaker.

[0053] Furthermore, the display unit 100 may be configured to highlight in real time the content of statements in which an emotional change has occurred, in response to the analysis by the emotion prediction unit 90. By configuring the automatic meeting minutes creation system 2 in this way, all attendees can check the timing of emotional changes during the meeting, which makes it easier to adjust the progress of the meeting and the focus of the discussion.

[0054] The display unit 100 may be configured to receive emotion data predicted by the emotion prediction unit 90 and to graph and display the changes in the attendees' emotions in real time. The graph displayed by the display unit 100 may be, for example, a bar graph or a pie chart, or a line graph to check changes over time.

[0055] <How the automated meeting minutes creation system works> Next, the operation of the automated meeting minutes creation system 2 according to this embodiment will be described. Figure 6 is a flowchart showing an example of the meeting minutes creation process performed by the automated meeting minutes creation system 2 shown in Figure 5. This meeting minutes creation process corresponds to the meeting minutes creation process in Embodiment 1, and is the process of step S13 of the overall process shown in Figure 2. Note that the overall process is the same as the operation of the automated meeting minutes creation system 1, so its explanation will be omitted.

[0056] In the meeting minutes creation process, first, the acquisition unit 20 acquires video and audio data from the meeting (step S41). The automatic meeting minutes creation system 2 determines, based on the acquired audio data, whether or not any attendees made a statement (step S42). If it is determined that no one made a statement, the processing flow returns to step S41, and steps S41 and S42 are repeated until a statement is made.

[0057] On the other hand, if it is determined that any of the attendees have made a statement, the processing flow proceeds to step S43, where the facial recognition unit 30 determines the speaker based on the video data through facial recognition (step S43).

[0058] Next, the video analysis unit 80 extracts at least one of the speaker's attitude, facial expression, or reaction during their speech, as determined by the facial recognition unit 30, based on the video data (step S44). Optionally, the video analysis unit 80 may also extract gestures made by the identified speaker during their speech based on the video data (step S45).

[0059] Furthermore, the video analysis unit 80 extracts the voice tone of the speaker determined by the facial recognition unit 30 during their speech based on the audio data (step S46).

[0060] Next, the emotion prediction unit 90 predicts the speaker's emotion based on at least one of the attitudes, facial expressions, reactions, and gestures extracted in this manner, and the determined speaker's voice tone (step S47). The emotion prediction unit 90 outputs the predicted speaker's emotion data to the minutes creation unit 40.

[0061] Next, the minutes creation unit 40 creates minutes that reflect the decided speakers, the content of their statements, and the predicted emotions of the speakers in real time (step S48). The minutes creation unit 40 outputs the created minutes data to the display unit 100.

[0062] Next, the display unit 100 displays the minutes received from the minutes creation unit 40 in real time (step S49). Here, the minutes creation unit 40 sequentially adds the speakers, the content of their statements, and the speakers' emotions to the minutes, and the display unit 100 displays the updated minutes in real time as the minutes are updated. Then, the automatic minutes creation system 2 terminates this minutes creation process. Similar to Embodiment 1, the automatic minutes creation system 2 repeatedly executes this minutes creation process until it determines that the meeting has ended.

[0063] Next, we will explain the emotion change display process. Figure 7 is a flowchart showing an example of the emotion change display process performed by the automated meeting minutes creation system 2 shown in Figure 5. The emotion change display process is performed when an attendee makes a statement and data related to that statement is extracted. In other words, the emotion change display process is performed after step S46 of the meeting minutes creation process shown in Figure 6.

[0064] In the emotion change display process, the emotion prediction unit 90 first predicts the emotion of the speaker based on the various data extracted in steps S44 to S46 of the meeting minutes creation process (step S51). This step S51 process corresponds to step S47 of the meeting minutes creation process.

[0065] Next, the emotion prediction unit 90 extracts the previously predicted emotion from the minutes created by the minutes creation unit 40 (step S52). Then, the emotion prediction unit 90 determines whether the emotion of the speaker predicted this time has changed from the emotion of the speaker predicted last time (step S53). If it determines that the speaker's emotion has not changed, the automatic minutes creation system 2 terminates this emotion change display process.

[0066] On the other hand, if the system determines that the speaker's emotions have changed, the emotion prediction unit 90 outputs information to the display unit 100 indicating this. The display unit 100 highlights the speaker's predicted emotions in real time, along with the speaker and the content of their remarks (step S54). The automated meeting minutes creation system 2 then terminates this emotion change display process. This emotion change display process is repeated until the meeting ends.

[0067] As described above, the automatic meeting minutes creation system 2 according to this embodiment further comprises a video analysis unit 80 that extracts at least one of the speaker's attitude, facial expression, or reaction during their speech, as determined by the facial recognition unit 30, based on the video data acquired by the acquisition unit 20, and an emotion prediction unit 90 that predicts the speaker's emotions based on at least one of the extracted attitude, facial expression, or reaction and a voice analysis of the speaker's speech (e.g., voice tone). The meeting minutes creation unit 40 is configured to reflect the speaker's emotions in the meeting minutes. By configuring the automatic meeting minutes creation system 2 according to this embodiment in this way, the speaker's emotions (emotional state) during the meeting can be grasped in real time and reflected in the meeting minutes. This allows the nuances of the speaker's speech during the meeting to be conveyed more accurately.

[0068] In this embodiment of the automated meeting minutes creation system 2, the emotion prediction unit 90 may be configured to make a positive or negative evaluation of the speaker's statements based on the speaker's statements and the predicted speaker's emotions. The emotion prediction unit 90 can output such evaluations to the meeting minutes creation unit 40. The meeting minutes creation unit 40 may be configured to add the positive or negative evaluation to the meeting minutes in correspondence with the statements. The display unit 100 can then display the positive or negative evaluation in real time. This allows the emotion evaluation to be reflected in the meeting minutes, resulting in the effect of making the content of the meeting minutes easier to understand.

[0069] Although not shown in the figures, the automated meeting minutes creation system 2 according to this embodiment may further include a providing unit that provides automated responses or feedback based on the content of the speaker's remarks and the predicted emotions of the speaker. Specifically, the providing unit analyzes the content of the speaker's remarks and the predicted emotions of the speaker, and provides appropriate automated responses or feedback as needed. The display unit 100 may be configured to display the automated responses or feedback provided by the providing unit in real time. Displaying appropriate automated responses or feedback in real time has the effect of improving the quality of the meeting.

[0070] In the embodiments described above, the disclosure has been described as a hardware configuration, but the disclosure is not limited thereto. The disclosure can also be implemented by having a computer program run on a processor in a computer to perform the processing of the automatic meeting minutes creation systems 1 and 2 described in the embodiments described above.

[0071] Finally, the hardware configurations of the automated meeting minutes creation systems 1 and 2 will be described. Figure 8 is a block diagram showing an example of the hardware configuration of the automated meeting minutes creation systems 1 and 2 of this disclosure. As shown in Figure 8, the automated meeting minutes creation systems 1 and 2 include a network interface 1000, a processor 2000, and memory 3000. The network interface 1000 may be used not only for the network described above but also for communicating with network nodes. The network interface 1000 may include, for example, a network interface card (NIC) compliant with the IEEE 802.3 series. IEEE stands for Institute of Electrical and Electronics Engineers.

[0072] The processor 2000 performs the various processes of the automated meeting minutes creation systems 1 and 2 described above by reading and executing software (computer programs) from memory 3000. The processor 2000 may be, for example, a microprocessor, an MPU (Micro-Processing Unit), or a CPU (Central Processing Unit). The processor 2000 may include multiple processors.

[0073] Memory 3000 is composed of a combination of volatile and non-volatile memory. Memory 3000 may also include storage located away from the processor 2000. In this case, the processor 2000 may access memory 3000 via an I / O (Input / Output) interface not shown.

[0074] In the example shown in Figure 8, memory 3000 is used to store a group of software modules. The processor 2000 can read these software modules from memory 3000 and execute them, thereby performing the various processes of the automated meeting minutes creation systems 1 and 2 described in the above embodiment.

[0075] As explained with reference to Figure 8, each of the processors in the above-described embodiment of the automatic meeting minutes creation system 1 and 2 executes one or more programs that include a set of instructions for causing the computer to perform the algorithm described with reference to the drawings.

[0076] Some or all of the processing in the above-described automated meeting minutes creation systems 1 and 2 can be implemented as computer programs. Such programs can be stored and supplied to a computer using various types of non-temporary computer-readable media. Non-temporary computer-readable media include various types of tangible recording media. Examples of non-temporary computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memory (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). Programs may also be supplied to a computer by various types of temporary computer-readable media. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. Temporary computer-readable media can be supplied to a computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

[0077] Although the present invention has been described with reference to embodiments, the present invention is not limited to the above embodiments and can be modified as appropriate without departing from the spirit of the invention.

[0078] Some or all of the above embodiments may be described as follows, but are not limited to the following. (Note 1) A registration section for registering one or more attendees who will be attending the meeting, An acquisition unit that acquires video and audio data during the aforementioned meeting, A facial recognition unit determines the speaker based on the aforementioned video data, A minutes preparation department prepares minutes based on the speakers and the content of their statements, If the speaker determined above is not one of the one or more attendees, the identification unit identifies the speaker as an unregistered attendee. A storage unit that stores the content of statements made by the identified unregistered attendee based on the face and clothing of the identified unregistered attendee, A determination unit that determines who the identified unregistered attendee is based on the video and audio data from the meeting, Equipped with, The minutes preparation unit updates the minutes based on the determination result of the determination unit. Automatic meeting minutes creation system. (Note 2) The determination unit obtains names from the audio data during the meeting that are not among the one or more attendees, and determines who the identified unregistered attendee is based on the obtained names. The automated meeting minutes creation system described in Appendix 1. (Note 3) The determination unit determines who the identified unregistered attendee is based on the content of statements made by other attendees before and after the statement made by the identified unregistered attendee. The automated meeting minutes creation system described in Appendix 1. (Note 4) A video analysis unit extracts at least one of the attitude, facial expression, or reaction of the determined speaker during their speech based on the aforementioned video data. An emotion prediction unit predicts the speaker's emotions based on at least one of the extracted attitudes, facial expressions, and reactions, and the voice analysis of the speaker's statements determined above. Furthermore, The minutes preparation unit reflects the predicted emotions of the speakers in the minutes. An automated meeting minutes creation system as described in any one of the items in Appendix 1 to 3. (Note 5) The emotion prediction unit analyzes the facial expressions and voice tone of the determined speaker during their speech to predict the speaker's emotions. The automated meeting minutes creation system described in Appendix 4. (Note 6) The video analysis unit extracts the gestures of the determined speaker in the video during the speech, The emotion prediction unit further predicts the speaker's emotion based on the gestures of the speaker that have been determined. The automated meeting minutes creation system described in Appendix 5. (Note 7) The system further includes a display unit that displays, in real time, the minutes prepared by the minutes preparation unit during the meeting, so as to include the predicted emotions of the speakers. The automated meeting minutes creation system described in Appendix 4. (Note 8) The emotion prediction unit determines whether the predicted emotion of the speaker has changed since the previous time. The display unit, upon determining that a change in emotion has occurred, highlights the content of the statement indicating the change in emotion in real time. The automated meeting minutes creation system described in Appendix 7. (Note 9) The system further includes a providing unit that provides an automated response or feedback based on the content of the speaker's statement determined above and the predicted emotions of the speaker. The automated meeting minutes creation system described in Appendix 4. (Note 10) The minutes preparation unit performs a positive or negative evaluation of the spoken content based on the content of the spoken speaker determined and the predicted emotions of the speaker, and adds the positive or negative evaluation to the minutes in correspondence with the spoken content. The automated meeting minutes creation system described in Appendix 4. (Note 11) The aforementioned minutes preparation department, The automatic minutes creation system described in Appendix 1, wherein, when the name of an identified unregistered attendee that could not be identified in the determination unit is entered, the system updates the information identifying the identified unregistered attendee to the entered name. (Note 12) Steps to acquire video and audio data during the meeting, Based on the aforementioned video data, the step of determining the speaker by facial recognition, The steps include creating minutes based on the speakers determined above and the content of their statements, If the speaker determined above is not one of the attendees, the step is to identify the speaker as an unregistered attendee, Based on the face and clothing of the identified unregistered attendee, the step of recording the content of what the identified unregistered attendee said, The steps include determining who the identified unregistered attendee is based on the video and audio data from the meeting, The steps include updating the minutes based on the determination result of the determination unit, This includes methods for automatically creating meeting minutes. (Note 13) On the computer, The process of acquiring video and audio data during the aforementioned meeting, Based on the aforementioned video data, a process is performed to determine the speaker by facial recognition, The process of creating minutes based on the determined speakers and the content of their statements, If the speaker determined above is not one of the attendees, the process of identifying the speaker as an unregistered attendee is performed. Based on the face and clothing of the identified unregistered attendee, the process involves recording the content of what the identified unregistered attendee said. A process to determine who the identified unregistered attendee is based on the video and audio data from the aforementioned meeting, Based on the determination result of the determination unit, the process of updating the minutes is performed. A program that executes something.

[0079] Some or all of the elements (e.g., configuration and function) described in Appendices 2 to 11 that are dependent on Appendice 1 may also be dependent on Appendices 12 and 13 in the same manner as those described in Appendices 2 to 11. Some or all of the elements described in any appendice may be applicable to various hardware, software, recording means, systems, and methods for recording software.

[0080] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure can be made as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate. [Explanation of Symbols]

[0081] 1, 2 Automatic meeting minutes creation system 10 Registration Department 20 Acquisition Department 30. Facial Recognition Section 40 Minutes Preparation Department 50 Specific section 60 Storage section 70 Judgment section 80 Video Analysis Department 90 Emotion Prediction Department 100 Display 1000 network interfaces 2000 processor 3000 memory

Claims

1. A registration section for registering one or more attendees who will be attending the meeting, An acquisition unit that acquires video and audio data during the aforementioned meeting, A facial recognition unit determines the speaker based on the aforementioned video data, A minutes preparation department prepares minutes based on the speakers and the content of their statements, If the speaker determined above is not one of the one or more attendees, the identification unit identifies the speaker as an unregistered attendee. A storage unit that stores the content of statements made by the identified unregistered attendee based on the face and clothing of the identified unregistered attendee, A determination unit that determines who the identified unregistered attendee is based on the video and audio data from the meeting, Equipped with, The minutes preparation unit updates the minutes based on the determination result of the determination unit. Automatic meeting minutes creation system.

2. The determination unit obtains names from the audio data during the meeting that are not those of the one or more attendees, and determines who the identified unregistered attendee is based on the obtained names. The automated meeting minutes creation system according to claim 1.

3. The determination unit determines who the identified unregistered attendee is based on the content of statements made by other attendees before and after the statement made by the identified unregistered attendee. The automated meeting minutes creation system according to claim 1.

4. A video analysis unit extracts at least one of the attitude, facial expression, or reaction of the determined speaker during their speech based on the aforementioned video data. An emotion prediction unit predicts the speaker's emotions based on at least one of the extracted attitudes, facial expressions, and reactions, and the voice analysis of the speaker's statements determined above. Furthermore, The minutes preparation unit reflects the predicted emotions of the speakers in the minutes. The automated meeting minutes creation system according to any one of claims 1 to 3.

5. The emotion prediction unit analyzes the facial expressions and voice tone of the determined speaker during their speech to predict the speaker's emotions. The automated meeting minutes creation system according to claim 4.

6. The video analysis unit extracts the gestures of the determined speaker in the video during the speech, The emotion prediction unit further predicts the speaker's emotion based on the gestures of the speaker that have been determined. The automated meeting minutes creation system according to claim 5.

7. The system further includes a display unit that displays, in real time, the minutes prepared by the minutes preparation unit during the meeting, so as to include the predicted emotions of the speakers. The automated meeting minutes creation system according to claim 4.

8. The emotion prediction unit determines whether the predicted emotion of the speaker has changed since the previous time. The display unit, upon determining that a change in emotion has occurred, highlights the content of the statement indicating the change in emotion in real time. The automated meeting minutes creation system according to claim 7.

9. Steps to acquire video and audio data during the meeting, Based on the aforementioned video data, the step of determining the speaker by facial recognition, The steps include creating minutes based on the speakers determined above and the content of their statements, If the speaker determined above is not one of the attendees, the step is to identify the speaker as an unregistered attendee, Based on the face and clothing of the identified unregistered attendee, the step of recording the content of what the identified unregistered attendee said, The steps include determining who the identified unregistered attendee is based on the video and audio data from the meeting, The steps include updating the minutes based on the judgment result, This includes methods for automatically creating meeting minutes.

10. On the computer, The process of acquiring video and audio data during a meeting, Based on the aforementioned video data, a process is performed to determine the speaker by facial recognition, The process of creating minutes based on the determined speakers and the content of their statements, If the speaker determined above is not one of the attendees, the process of identifying the speaker as an unregistered attendee is performed. Based on the face and clothing of the identified unregistered attendee, the process involves recording the content of what the identified unregistered attendee said. A process to determine who the identified unregistered attendee is based on the video and audio data from the aforementioned meeting, Based on the judgment result, the process of updating the minutes is performed, A program that executes something.

Images