Systems and programs for meeting evaluation

Abstract

To provide a system and program for evaluating the productivity of a meeting without relying on arbitrary human decisions. [Solution] The system includes a user profile storage unit, a timing unit that acquires time data related to the date and time of the meeting, a recording unit that acquires meeting record data including video or audio from the meeting, a record division unit that divides the meeting record data into multiple segments associated with any of multiple users and stores them, and the multiple segments are associated with time data, and a scoring unit that analyzes the multiple segments and quantifies and scores the productivity of the meeting, and the scoring unit quantifies the productivity based on at least one of the following: the degree of relevance of comments related to the purpose or agenda of the meeting in a first period near the start of the meeting, the degree of relevance of comments related to discussions or collection of questions and answers about the meeting in a second period near the end of the meeting, the frequency of occurrence of taboo phrases, and statistical information on the duration of comments.

Description

[Technical Field]

[0001] The present invention relates to a system and a program for evaluating a meeting. [Background technology]

[0002] With the spread of online communication systems, it has become common to hold online meetings involving multiple users and automatically compile the transcripts and minutes of those meetings using software. Various attempts to improve the efficiency of online meetings have also been proposed.

[0003] Patent Document 1 discloses a conference support system for supporting the management of conferences in an organization, which includes a videoconferencing device that enables online videoconferencing between multiple locations, a speech recognition unit that acquires voice data of each attendee using the videoconferencing device and identifies the speaker of each statement in the voice data based on a comparison between the voice data and pre-registered voice characteristic information of each attendee, and a timeline management unit that outputs the voice data of each attendee acquired by the speech recognition unit as a timeline in chronological order of the statements, thereby capitalizing the content of the conference and improving the efficiency of the management of the conference. Patent Document 1 also discloses a function that performs various evaluations and scoring of the content of the conference in real time or after the fact. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-061594 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the above-mentioned conventional technologies, it is not clear how to ensure the reliability of the evaluation and scoring of the content of a meeting. In other words, the conventional technologies have not been able to solve the problem that the evaluation and scoring of meetings are ultimately determined arbitrarily by humans. [Means for solving the problem]

[0006] In view of the above-mentioned problems, the embodiments of the present invention can provide the following.

[0007] Aspect 1. A system for evaluating a conference by a plurality of users by a computer, comprising: a user profile storage unit that stores a plurality of user profiles for each of the plurality of users; a timekeeping unit that acquires time data relating to the date and time of the meeting; a recording unit that acquires conference record data including at least video or audio of the conference in association with the time data; a record division unit that divides the conference record data into a plurality of segments, each associated with one of the plurality of users based on the plurality of user profiles, and stores the divided segments, and each of the plurality of segments is associated with the time data; a scoring unit that analyzes the plurality of segments and scores the productivity of the meeting by quantifying it; Including, The scoring section is the relevance of statements to the purpose or agenda of the meeting during a first period of time near the start of the meeting; the relevance of the statements to the discussion or collection of questions and answers regarding the meeting during a second period near the end of the meeting; The frequency of occurrence of taboo words during the meeting; and Statistical information about the speaking times of the plurality of users during the conference The method is configured to perform quantification based on at least one of the following points: A system characterized by:

[0008] Aspect 2. The system of aspect 1, wherein the scoring unit is configured to perform a numerical evaluation with different weightings for each of the aspects.

[0009] Aspect 3. The system according to any one of aspects 1 and 2, wherein the recording unit is configured to record at least audio related to the conference, and the conference record data includes transcription data obtained by recognizing the audio.

[0010] Aspect 4. a minutes creation unit that creates minutes data of the meeting based on the meeting record data and / or the plurality of segments. The system according to any one of aspects 1 to 3, further comprising:

[0011] Aspect 5. The system according to any one of aspects 1 to 4, wherein the scoring unit is configured to create feedback data in a format that allows feedback, based on the result of the digitization.

[0012] Aspect 6. The system described in any one of aspects 1 to 5, wherein the scoring unit is configured to accept and store data regarding the type of meeting, language, or location prior to the meeting being held, and to take the data regarding the type of meeting into consideration when quantifying the data.

[0013] Aspect 7. The system according to any one of aspects 1 to 6, wherein the digitization by the scoring unit includes a first digitization for the entire conference and a second digitization for each of the plurality of users.

[0014] Aspect 8. The system according to any one of aspects 1 to 7, wherein each of the plurality of user profiles is created based on one or more of a message history, an input history, and a created document of each of the plurality of users.

[0015] Aspect 9. 1. A computer program for evaluating a conference of multiple users, comprising: storing a plurality of user profiles for each of the plurality of users; obtaining time data relating to the date and time of the meeting; acquiring conference record data including at least video or audio from the conference in association with the time data; dividing the conference record data into a plurality of segments, each associated with one of the plurality of users based on the plurality of user profiles, and storing the divided segments, and each of the plurality of segments being associated with the time data; analyzing the plurality of segments and quantifying and scoring the productivity of the meeting; Including, The scoring step includes the following steps: the degree of purpose or agenda setting for the meeting during a first period of time near the start of the meeting; the extent of discussion or question and answer session regarding the conference during a second period near the end of the conference; The frequency of occurrence of taboo words during the meeting; and Statistics of speaking time of the plurality of users during the conference The method is configured to perform quantification based on at least one of the following points: A computer program comprising: [Effects of the Invention]

[0016] According to the present invention, an effect is obtained in that the contents of a conference can be objectively evaluated without relying on arbitrary human judgment. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 2 is a block diagram illustrating the operation flow of the system according to the present invention. [Figure 2] 2 shows an example of an operation flow of the system according to the first embodiment of the present invention. [Figure 3] 10 shows an example of a user interface that can be used in the first embodiment and allows a user to input a conference type selection. [Figure 4] Examples of scoring items that can be used in Example 1 are shown below. [Figure 5] 10 shows an example of a weighting definition table that can be used in the first embodiment. [Figure 6] 10 shows an example of an output of a scoring result created by the system according to the first embodiment. [Figure 7] 10 shows an example of an output of a scoring result created by a system according to a modification of the first embodiment. [Figure 8] 10 shows an example of an operation flow of a system according to a second embodiment of the present invention. [Figure 9] An example of a prompt that can be used in Example 2 is shown below. [Figure 10] 10 shows an example of an output of a scoring result created by the system according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] The system according to the present invention is a system that can be implemented on a computer and may be implemented on multiple computers connected via a network, on the server side, or on a client terminal side. Alternatively, the system can be implemented on a standalone computer. In this specification, the term "computer" refers to a computer having a general hardware configuration, primarily including a processor, a storage device, a communication interface, etc.

[0019] The system according to the present invention is configured to evaluate a conference held by multiple users, and the type of the conference does not matter as long as data related to the conference is available. For example, an online conference held using some existing conference system may be evaluated by acquiring data in real time, or recorded data may be used after the conference. Alternatively, a completely offline conference or a hybrid conference in which some participants are online may be evaluated by recording data in real time or after the conference.

[0020] The system mainly includes a user profile storage unit, a timing unit, a recording unit, a record division unit, and a scoring unit, and these components may be modules implemented as software, and these modules may be stored in a single storage device or may be distributed and stored in multiple storage devices (these storage devices may be local or on the cloud). Alternatively, some or all of the components may be implemented as hardware.

[0021] FIG. 1 is a block diagram illustrating the operational flow of the system according to the present invention. First, in step S101, a user profile for identifying each user who is a conference participant is created and saved. The user profile is a data set containing information about each user, and may include, for example, an ID (unique ID), name, age, facial photo, organization affiliation, email address, and information about various web service accounts. This user profile does not necessarily need to be a data set that can completely identify individual users at the time the conference is held; it may simply contain a provisionally registered ID. In other words, it is sufficient to have information that can associate comments made in the conference with that user.

[0022] The provisional user profile may be later converted into a complete user profile by including identification data entered by the user. In some embodiments, a user profile may be created for a user from the user's speech history, input history, and documents created.

[0023] In step S102, time data related to the date and time of the conference is acquired and managed using a known clock, an NTP server, or the like. The time data may include information about the start date and time of the conference, the end date and time of the conference, and the duration of each speech in the conference (such as the time from the start to the end of the speech, and / or the start and end times of the speech). In particular, if the conference is held across multiple time zones, it is preferable to use time standardized according to some standard (such as UTC).

[0024] In step S103, conference record data associated with the time data is acquired. The conference record data includes at least video or audio from the conference. The conference record data can be acquired using known devices such as a camera or microphone, and is data that can identify the user who spoke. To identify the user who spoke, known methods can be used, such as identification from a user ID / account on an online conference system, identification by facial recognition, or identification by voiceprint.

[0025] If the recording includes audio, it is preferable to include transcription data of the audio in the conference record data in consideration of data availability. Alternatively, if the recording includes video, it is preferable to include data (which may include images, video, audio, or text) that recognizes information such as sign language or a written presentation that can be recognized from the video. Known voice recognition engines and image recognition engines can be used for transcription and information recognition. Note that in this specification, the term "utterance" is not necessarily limited to audio utterances, but is a concept that also encompasses visual utterances such as sign language and written communication.

[0026] In some embodiments, user identification and utterance recognition may use a large-scale language model (LLM) trained on the user, either in real time during the meeting or pre-trained using data related to the user's past participation in speech, typing, or written documents.

[0027] In step S104, the conference record data obtained above is divided for each speaking user based on the user profile and saved as segments associated with time data (such as the time from the start to the end of the speech and / or the start and end times of the speech). For example, a certain segment can be associated with the time that user A spoke from 0:20:15 to 0:22:27. Each segment may be tagged with a unique ID based, for example, on the start time of the speech.

[0028] Dividing the conference record data into segments in this way has the effect of making it easier to perform the scoring evaluation described below. Furthermore, when multiple users make overlapping speeches, the segments may overlap in time. Separating overlapping speeches from multiple users can also be achieved by using the recognition engine processing described above. If separation is not possible, the segment can be discarded from processing as indecipherable noise.

[0029] In step S105, the above segments are analyzed to quantify and score the productivity of the meeting. This quantification is characterized by the fact that it is possible to automatically perform objective evaluation by a machine by performing processing based on each perspective described below. Note that in this specification, the term "quantification" does not necessarily mean that the numerical value is what humans see. It should be noted that as long as a computer calculates quantitative values ​​in this processing, even if the output does not appear to be numerical, such as an "A rating" or "B rating," the processing is included in the concept of "quantification."

[0030] The system according to the present invention may further include a minutes-creating unit that creates (documents) minutes data of the meeting based on the meeting record data or segments. Such minutes can be saved in any format and provided to users or administrators.

[0031] The operation of the scoring unit of the system according to the present invention will be described in more detail below. The scoring unit can quantify the productivity of a meeting based on one or more or all of the following four perspectives. The inventors discovered that there are certain commonalities in the activities known as meetings and incorporated these into the present invention. That is, the inventors discovered that whether a meeting is "productive" or "unproductive" is not merely an individual's opinion, but is a concept that can be generalized and evaluated. In other words, this quantification is based on insight and analysis of the collective behavior of a group of meeting participants (a group of human users), a calculation that can be automated by machine, and is based on scientific concepts. Perspective 1: The relevance of statements to the purpose or agenda of the meeting during the first period around the start of the meeting. Point 2: The relevance of statements to the discussion or question and answer session regarding the conference during the second period near the end of the conference. Viewpoint 3: Frequency of taboo words during the meeting Viewpoint 4: Statistics of user speaking time during the meeting

[0032] [Perspective 1: The degree of relevance of statements regarding the purpose or agenda of the meeting during the first period around the start of the meeting] For a meeting to be highly productive, it is important to clarify the purpose (goal) or agenda of the meeting. It is also important that this clarification occurs near the beginning of the meeting. If the purpose is presented later in the meeting, participants generally tend to have already lost interest, and the meeting will likely end in futility.

[0033] In the present invention, a first period is set near the start of the meeting (preferably a period that is 10% of the entire meeting duration, more preferably 7%, and even more preferably 5% of the entire meeting duration). The start of this first period is preferably aligned with the start of the meeting, but it is acceptable to set it slightly later than the start of the meeting depending on the circumstances. During this first period at the beginning of the meeting, the number of times that words related to the "purpose" or "agenda" of the meeting are uttered by the same user within the same segment (within the same utterance) as words related to the "meeting" can be used as a parameter indicating the degree of relevance of the utterances related to the purpose or agenda of the meeting. The definition of "within the same utterance from the same user" is intended to eliminate noise that is unrelated to the purpose of the meeting, even if those words are uttered in different contexts.

[0034] More specifically, the degree of relevance can be calculated by counting the number of times the words "purpose" or "agenda" or their synonyms (e.g., "goal" or "agenda") and "meeting" or their synonyms (e.g., "place") are uttered by the same user in the same segment (within the same utterance). Hereinafter, "synonym" is considered to encompass "homonyms." Synonyms can be quantitatively calculated based on the semantic distance of linguistic word vectors using existing natural language processing technologies such as Word2Vec and BERT via programming languages ​​such as Python (calculation may also be performed taking into account an existing thesaurus dictionary). For example, synonyms can be defined herein as a cosine similarity between a reference word and a target word being measured when Word2Vec is used, with a cosine similarity of 0.9 or greater, more preferably 0.95 or greater. This calculation can be performed based on a bigram model. Although this specification uses a Japanese-language conference as an example, the present invention is not limited to this. By calculating the semantic distance of various languages ​​using any language corpus, the present invention can also be applied to conferences in languages ​​other than Japanese or multilingual conferences. In one embodiment, a segment containing a target word may be input to an LLM that has learned the semantic distance of each word in Japanese or another natural language, and the LLM may quantify whether the target word is a synonym of a reference word. Please note that the following explanations (examples of words) in Japanese can be substituted for various foreign languages ​​and applied to the present invention.

[0035] In some embodiments, "purpose" and "agenda" may be evaluated separately. That is, a parameter that depends on the number of times "purpose" or a synonym thereof and "meeting" or a synonym thereof are uttered by the same user in the same segment may be considered as the meeting purpose clarity, and a parameter that depends on the number of times "agenda" or a synonym thereof and "meeting" or a synonym thereof are uttered by the same user in the same segment may be calculated and evaluated separately as the agenda clarity.

[0036] [Point 2: The relevance of statements to the discussion or question and answer session regarding the conference during the second period near the end of the conference] For a meeting to be highly productive, it is important to make a statement that the meeting has been properly concluded near the end. This is because it has been observed that it is psychologically important not to give participants the impression that the meeting ended with no conclusions reached.

[0037] This aspect 2 preferably mainly consists of one or both of two points: whether the purpose or agenda has been collected (in relation to aspect 1) and whether the meeting has been concluded.

[0038] To quantify whether a purpose or agenda was recovered, we first list the other words that appear in the same segment where purpose or agenda or its synonyms coexist with "discussion," "about," "regarding," or their synonyms (connecting words) during the first period mentioned above. After removing fillers, interjections, and other noise from these words, we set the candidate words for purpose or agenda.

[0039] Then, a second period is set near the end of the meeting (preferably a period that is 95% of the length of the entire meeting, more preferably 75%, and even more preferably 50% of the length). The end of this second period is preferably set to coincide with the end of the meeting, but it is acceptable for it to end slightly before the end of the meeting depending on the situation. It is preferable to set the second period longer than the first period, as this makes it easier to fully analyze the content of the discussion. In one embodiment, the entire meeting period may be broadly divided into only the first period and the second period. In another embodiment, the entire meeting period may include periods other than the first and second periods (which may be breaks).

[0040] In such a second period at the end of the meeting, a parameter that depends on the number of times the above-mentioned purpose / agenda candidate words appear again can be defined as a parameter (progress management coverage) indicating whether the purpose or agenda of the meeting has been collected.

[0041] In some embodiments, progress management coverage may be calculated based on the recovery rate of purpose / agenda candidate words. For example, if multiple candidate words appeared in the first period, the percentage of those that reappeared in the second period can be calculated to calculate the parameter.

[0042] Furthermore, the user who uttered the purpose / agenda candidate words in the first period at the beginning of the meeting and the user who uttered the purpose / agenda candidate words again in the second period at the end of the meeting do not necessarily have to be the same user. In some embodiments, the progress management coverage may be calculated on the condition that both are uttered by the same user.

[0043] In another embodiment, the number of times that a word corresponding to the purpose / agenda candidate word was included in the same segment during the second period for the richness of the issue, clarity of the rationale, degree of consensus, clarity of the decision, or clarity of the ToDo may be calculated.

[0044] The degree of thoroughness of the issues indicates whether the issues that needed to be answered in the meeting were clearly discussed, and can be calculated, for example, as the number of times the word "issue" or a synonym thereof was used.

[0045] Clarity of evidence indicates whether opinions were expressed from an objective perspective based on evidence, data, examples, etc. in the meeting, and can be calculated, for example, as the number of times the words "data," "examples," "evidence," or similar words were used.

[0046] The degree of consensus indicates whether the main points of discussion in a meeting were not left ambiguous and whether a consensus was reached that leads to the next action. For example, it can be calculated as the number of times that words such as "decision," "agreement," "ToDo," or similar words were used.

[0047] Decision clarity indicates whether it is clear what decisions were made in a meeting and why, and can be calculated, for example, as the number of times that "decision," "reason," "basis," or synonyms thereof are used. In some embodiments, the number of times that "reason," "basis," or synonyms thereof appear in a certain number of segments going back from the segment in which "decision" or a synonym thereof appeared in the second period (for example, from the segment immediately preceding that segment to the segment five segments before that) may be counted and used as a parameter. This is based on the idea that the appearance of such words within a certain proximity of each other contributes to improving the productivity of meetings.

[0048] ToDo clarity indicates whether specific actions, responsible parties, and deadlines were clearly set in a meeting, and can be calculated, for example, as the number of times "ToDo," "things to do," "responsible party," "deadline," "action," or synonyms thereof were uttered. In some embodiments, the number of times "responsible party," "deadline," "action," or synonyms thereof appeared in segments a certain number of segments after the segment in which "ToDo," "things to do," or synonyms appeared in the second period (for example, from the segment immediately following that segment to the segment three segments later) may be counted and used as a parameter. This is based on the idea that the appearance of such words within a certain proximity of each other contributes to improving meeting productivity.

[0049] There are various methods for calculating these parameters, and a person skilled in the art who has read this specification will be able to understand what calculation method can be used to implement the present invention based on the above-mentioned viewpoints.

[0050] The quantification of whether a meeting has been summarized can be calculated as the number of times "question," "doubt," "wrap-up," "summary," or similar words are used during the second period. The second period used in this quantification may be shorter, for example, 10% of the total meeting duration, or more preferably 5%. In some embodiments, these may be further calculated separately as the Q&A richness and wrap-up degree.

[0051] The Q&A richness indicates whether there are any questions or uncertainties during the discussion in the meeting, and whether the meeting participants have confirmed them. It can be calculated based on whether a segment related to another user (assumed to be a respondent) appears immediately after a segment containing "question," "confusion," or synonyms thereof in the second period. This parameter is set in consideration of the fact that, generally, when no questions are asked, segments by the same user tend to appear with silence in between.

[0052] The wrap-up level indicates whether participants are making sure there is no misunderstanding regarding the decisions and tasks to be completed in the meeting, and can be calculated based on whether a segment containing "wrap-up," "summary," or a synonym thereof, related to another user, appears after a segment containing "it's okay," "got it," or a synonym thereof, in the second period. This parameter is set in consideration of the phenomenon commonly observed as a dialogue ritual in meetings.

[0053] [Perspective 3: Frequency of taboo phrases during the meeting] For a meeting to be highly productive, it is important that participants avoid excessive attacks on other participants and that their enthusiasm is not diminished. The parameter related to this aspect 3 can be calculated by maintaining a list of taboo words (a list that can be created from a corpus of known abusive words, vulgar words, negative words, etc.) in the system and depending on the frequency of occurrence of the taboo words or their synonyms. For example, taboo words could be "no" or "impossible." In some embodiments, the parameter can be calculated by comparing the ratio of segments that do not contain taboo words to the ratio of segments that contain "proposal," "idea," or their synonyms among the total number of segments contained in the meeting recording data. This serves as an evaluation of whether constructive discussion is being hindered by taboo words.

[0054] [Viewpoint 4: Statistics on user speaking time during the meeting] For a meeting to be highly productive, it is important that only a few participants do not monopolize long speaking times. This serves as an indicator of whether the other participants are losing motivation. The parameter related to this viewpoint 4 can be calculated by extracting consecutive segments from the meeting recording data that are associated with the same user, calculating the number of segments contained in those whose total duration exceeds a threshold (e.g., one minute), and calculating the ratio to the total number of segments. This process allows for appropriate extraction of cases where the same user is speaking over and over again, even if the segments are separated.

[0055] The scoring unit may perform calculations using one or more of the four perspectives described above, or may perform calculations using all of them. When performing calculations using multiple perspectives, the scoring unit may assign different weights to each perspective, quantify them, and calculate the sum of the weights.

[0056] In some embodiments, the scoring unit may receive and store data prior to a meeting regarding the type of meeting (e.g., whether it is a corporate business meeting, research forum, or club; or the purpose of the meeting, such as decision-making, problem-solving, education, or information sharing), the language type (including Japanese or other natural languages, as well as non-verbal communication such as sign language that can be recorded), or the location (e.g., the country or region where the meeting is held, or whether it is online or offline). This data may then be used in the calculation of the meeting's productivity (e.g., by changing the weighting of the four aspects described above depending on the data). For example, in the case of a hybrid meeting, the weighting may be determined based on the ratio of the number of users participating online to the number of users participating offline.

[0057] In another embodiment, which of the above four perspectives to adopt may be determined based on this data. For example, if the type of meeting is not for business purposes, such as a club, it is possible to consider redundant comments as acceptable and not quantify the productivity of the meeting based on perspective 4 (user speaking time).

[0058] In some embodiments, the scoring unit may quantify the productivity of a meeting by providing a first quantification for the entire meeting and a second quantification for each user. For example, the first quantification may calculate the productivity of the entire meeting, while the second quantification may perform calculations specific to individual users. For example, the second quantification may be performed by weighting segments associated with each user based on their user profile. For example, the user profile may store data related to past scoring results by the system and data related to past meeting participation records, and the latest scoring may include processing based on those past scoring results. For example, processing such as adding or subtracting a portion of past scores, calculating a moving average, or multiplying by a factor based on the number of participations may be possible.

[0059] The operations of this system described above can be implemented by a computer program. The program can be stored in storage, loaded into memory, and executed on a CPU to perform each step of the operations described above. [Example]

[0060] Example 1: Example of weighting based on the type of meeting purpose The first embodiment shows an example of quantifying meeting productivity by weighting according to the type of purpose of the meeting. Fig. 2 is a block diagram illustrating the operation flow of the system according to the first embodiment. In the first embodiment, a meeting including decisions and ToDos is set as an example.

[0061] In the first embodiment, the audio of the conference is first recorded (210), and then the recording unit saves the audio as transcribed data (212). An existing transcription API, such as Google Speech to Text, can be used for this transcription.

[0062] The obtained transcription data (214) is input to a record division unit together with a command (220) to create a segment, and a segment is created (222). At the same time, a minutes creation unit may also create minutes. In this way, segments / minutes containing utterances related to decisions and to-do items in the meeting content are obtained (224). For example, the transcription data (214) may be input to an API of a generation AI, which is an LLM, together with a prompt (220), and segments and minutes are obtained (224).

[0063] Data (230) on the pre-set meeting type is input into the scoring unit, and based on that, the productivity of the meeting is quantified for each of the above-mentioned perspectives (232). Figure 3 shows an example of a screen display (user interface) that allows the user to input the selection of the meeting type. In the example of Figure 3, the type of meeting can be selected based on the axes of emphasis on judgment / emphasis on information sharing and emphasis on one-to-many communication (1:N) / many-to-many communication (N:N). In the example display of Figure 3, a "decision-making" type meeting has been selected.

[0064] Returning to Figure 2, data on the weighting definitions (234) for each perspective according to the preset conference type is input to the scoring unit, which then averages the weights for each perspective and calculates the overall score (236). Figure 4 shows an example of items used for scoring. In this example, the above-mentioned perspective 1 corresponds to items 1.1 and 1.2. Perspective 2 corresponds to items 2.1, 2.2, 2.3, 3.1, 3.2, 3.3, 4.1, and 4.2. Perspective 3 corresponds to item 5.2, and perspective 4 corresponds to item 5.1.

[0065] Figure 5 shows an example of a weighting definition table. Weighting can be set for each item shown in Figure 4 according to the selection of the four types of meetings shown in Figure 3. Based on these definitions, the scoring unit can calculate the overall score.

[0066] Returning again to Figure 2, the resulting total score can be saved or sent for external output (240). Figure 6 shows an example of the scoring results that can be output in this way (including the total score and the scores for each item). This output can be sent to the user in any format.

[0067] As a variant, the scoring process can be further divided for each user, and some weighting can be applied to create a scoring result for each user. Figure 7 shows an example of a scoring result that can be created for a specific user in this way. In this example, the user's pages are displayed separately by tab.

[0068] Example 2: Example with feedback output In the second embodiment, in addition to quantifying the productivity of a meeting, improvement advice for improving productivity is also output as feedback to the user. Fig. 8 is a block diagram illustrating the operation flow of the system according to the second embodiment, which is generally the same as Fig. 2 of the first embodiment, except that an advice generation prompt is prepared (840), the scoring unit inputs the scoring results and segments (or minutes) to the LLM, which outputs improvement advice (842), and the scoring results and improvement advice are saved and transmitted (850).

[0069] FIG. 9 shows an example of an advice generation prompt (840). While the prompt can be written in any format, it is preferable to write it in the Markdown format, which specifies the item, as in this example. In some embodiments, the advice generation prompt (840) can be automatically generated by the system of the present invention. For example, the prompt (840) can be automatically generated by extracting one or more or all of the phrases, user names, and time data contained in the segment or minutes (824) created in step (822) and inputting them so that they fit into a pre-prepared prompt template. By inputting the prompt (840) created in this way into the LLM, the improvement advice output (842) clearly indicates which statements made by whom and at what time in the meeting were used as the basis for the scoring unit's scoring and / or were displayed in the scoring results created by the scoring unit.

[0070] 10 shows an example of the output obtained in Example 2. Along with the numerically converted scoring results, it is possible to include advice on how to improve the corresponding content as feedback.

Claims

1. A system for evaluating a conference by a plurality of users by a computer, comprising: a user profile storage unit that stores a plurality of user profiles for each of the plurality of users; a timekeeping unit that acquires time data relating to the date and time of the meeting; a recording unit that acquires conference record data including at least video or audio of the conference in association with the time data; a record division unit that identifies video or audio related to each user based on the plurality of user profiles, divides the conference record data into a plurality of segments each associated with one of the plurality of users, and stores the divided segments, and each of the plurality of segments is associated with the time data; a scoring unit that analyzes the plurality of segments and scores the productivity of the meeting by quantifying it; Including, each of the plurality of user profiles is configured to include data regarding past meetings of the corresponding user; The scoring section is the relevance of statements to the purpose or agenda of the meeting during a first period of time near the start of the meeting; the relevance of the statements to the discussion or collection of questions and answers regarding the meeting during a second period near the end of the meeting; The frequency of occurrence of taboo words during the meeting; and Statistical information about the speaking times of the plurality of users during the conference and The scoring unit is configured to perform a weighted numerical evaluation based on data relating to each user's past meetings. A system characterized by:

2. The system according to claim 1 , wherein the scoring unit is configured to perform a numerical evaluation with different weightings for each of the viewpoints.

3. The system of claim 1 , wherein the recording unit is configured to record at least audio related to the meeting, and the meeting recording data includes transcription data that recognizes the audio.

4. a minutes creation unit that creates minutes data of the meeting based on the meeting record data and / or the plurality of segments. The system of claim 1 further comprising:

5. The system of claim 1 , wherein the scoring unit is configured to create feedback data in a form that allows feedback based on the results of the quantification.

6. The system of claim 1, wherein the scoring unit is configured to receive and store data regarding the type, language, or location of the meeting prior to the meeting being held, and to take the data regarding the type of meeting into account when quantifying the data.

7. The system of claim 1 , wherein the scoring unit provides a numerical value for the entire conference and a second numerical value for each of the plurality of users.

8. The system according to claim 1 , wherein each of the plurality of user profiles is created based on one or more of a speech history, an input history, and a created document of each of the plurality of users.

9. 1. A computer program for evaluating a conference of multiple users, the computer comprising: storing a plurality of user profiles for each of the plurality of users; obtaining time data relating to the date and time of the meeting; acquiring conference record data including at least video or audio from the conference in association with the time data; dividing the conference record data into a plurality of segments, each associated with one of the plurality of users, by identifying video or audio related to each user based on the plurality of user profiles, and storing the divided segments, and each of the plurality of segments being associated with the time data; analyzing the plurality of segments and quantifying and scoring the productivity of the meeting; configured to cause each of the plurality of user profiles is configured to include data regarding past meetings of the corresponding user; The step of scoring includes the following steps: the degree of purpose or agenda setting for the meeting during a first period of time near the start of the meeting; the extent of discussion or question and answer session regarding the conference during a second period near the end of the conference; The frequency of occurrence of taboo words during the meeting; and Statistics of speaking time of the plurality of users during the conference and The step of scoring is configured to perform a weighted numerical evaluation based on data regarding each user's past meetings. A computer program comprising:

Images