Meeting Evaluation System

Abstract

To provide a meeting evaluation system for conducting just the right number of meetings on-site. [Solution] The meeting evaluation system 1 records audio data of a meeting on a communication device 3 held by a person in charge at a work site 2, and sends the audio data recorded on the communication device to an unsafety detection PF 4 on the cloud, and the unsafety detection PF, upon receiving the audio data, sends the audio data to a text conversion service 5 on the cloud for text conversion, and the text conversion service on the cloud sends the converted text data to the unsafety detection PF, which sends the text data received by the unsafety detection PF, along with a definition prompt consisting of meeting evaluation items and score evaluation criteria for each item, to a generation AI 5 on the cloud, and the generation AI performs a meeting evaluation with comments and a score based on the meeting evaluation items and the evaluation criteria for each item from the character strings present in the text data based on the definition prompt.

Description

[Technical Field]

[0001] The present invention relates to a meeting evaluation system that checks and evaluates whether pre-work meetings held at work sites as part of normal work are being conducted appropriately, immediately notifies the results to the on-site work supervisor, and shares the results of the check and evaluation so that they can be used at other sites. [Background technology]

[0002] At construction sites and various other construction projects, a meeting known as a morning assembly (hereinafter referred to as either a "meeting" or a "morning assembly," but these have the same meaning) is usually held before work begins, where the workers involved in the day's work are confirmed, the work process for each worker is confirmed, and any precautions specific to the site are communicated. However, confirmation of workers, work process confirmation, and communication of precautions vary widely from site to site, and there are many cases where confirmations or communication are missed.

[0003] The applicant of the present application has already proposed a technique disclosed in Japanese Patent No. 7337772 as a technique for ensuring that confirmation items and notices are conveyed in meetings at work sites of this kind. The disclosure of Patent No. 7337772 relates to an invention titled "Meeting Confirmation and Evaluation System," which states, "Provides a meeting confirmation and evaluation system that creates meeting minutes (hereinafter also referred to as "daily report") from recording files of meetings such as morning assemblies held at various work sites, confirms, quantifies, and evaluates various matters that must be regularly confirmed at the work site based on pre-created teacher data, and displays the evaluation results (evaluation scores and score distribution) on a monitor in the business headquarters that oversees the work site, and outputs the results in a list, etc., enabling information sharing between site supervisors including the general manager." In the problem to be solved by the invention (see paragraph 0006 of the specification of the same publication), the patent application states that "the invention comprises a work management system that stores detailed work data at construction sites and other work sites in advance, a server that stores training data including a pre-written industry glossary and a DB of various construction glossaries and a DB of dangerous work countermeasures, a transcription processing unit that converts audio files into text, a natural language processing unit that extracts sentences from the text-written conference audio files that can be used to appropriately evaluate the conference, and an internal processing unit that determines whether or not there are "items that must be confirmed" within the conference based on the training data. a step of transmitting a signal linked to detailed work data together with a recorded audio file of a meeting held at a work site in a work management system; a step of transmitting the detailed work data accumulated together with the audio file of the meeting to the computer in the work management system; a step of converting the transmitted audio file into text in the computer; a step of extracting target words or sentences by comparing the converted audio file data and detailed work data with teacher data consisting of an industry glossary, a DB of various construction terminology, and a DB of dangerous work countermeasures that have been previously converted into text; a step of recognizing necessary keywords by comparing the converted conference audio file data, detailed work data, and teacher data, and detecting countermeasures for dangerous processes and dangerous points; a step of identifying terms included in the teacher data from the conversations and communications during the conference at the work site and creating minutes of the conference in chronological order; and a step of determining whether or not there are "items that must be confirmed" at the conference held at the work site from the converted conference audio file data and detailed work data based on the teacher data.By configuring the system to include a step of quantifying the results of determining whether or not there are "items that must be confirmed" at meetings held on-site, and a step of comprehensively evaluating whether or not the confirmations and communication at the meetings were properly carried out based on the total of the quantified items (see the description of claim 1 of the patent), "meeting minutes (diary report) are automatically created, and the system quantifies and evaluates whether or not the items that must be confirmed, etc. at the work site are properly carried out based on a database (hereinafter referred to as "DB") that serves as pre-created teacher data, and if the content of the meeting on that day is determined to be insufficient based on the evaluation results (evaluation scores and score distribution), immediate feedback is provided to the site, and additional or supplementary instructions are given to make up for the deficiencies and promote safe work, etc., and the information is displayed on a monitor in the business headquarters that oversees the work site, making it possible to share information with site personnel including the general manager, so that the construction manager or other construction personnel can provide appropriate warnings and safety guidance to the work team without any omissions based on the evaluation results, and the evaluation results are also provided to subcontracting companies for the construction, etc. at predetermined intervals (for example, monthly). This will have the effect of "disclosing the information to the general public, improving the leadership of frontline team leaders and building stronger cooperative relationships on-site with all partner companies" (see paragraph 0008 of the specification of the same publication).

[0004] Figure 3 is a diagram showing an overall outline of the implementation of a meeting confirmation and evaluation system according to one embodiment of the disclosed invention, which is attached as Figure 1 in Japanese Patent Publication No. 7337772. In Figure 3, reference numeral 101 denotes the meeting confirmation and evaluation system according to the first embodiment of the disclosed invention, 102 denotes a work site, 103a denotes a meeting audio file, 103b denotes an instruction signal, 103c denotes detailed work data, 104 denotes a work management system, 105 denotes a computer, 100 denotes a monitor, and 107 denotes an output result (note that the reference numerals have been changed to three digits by the applicant of this application to clarify that they are prior art).

[0005] However, the invention disclosed in Japanese Patent No. 7337772 (see Figure 3 and elsewhere) has the following drawbacks and problems: (1) In the invention disclosed in Japanese Patent No. 7337772, various construction sites are involved, and the terminology used varies depending on the type of construction work. Furthermore, even the same terminology may be expressed differently at each site. Therefore, a database must be created in advance, manually converting past meeting audio and interviews with each worker into text data (see paragraph 0016 of the specification of the same patent application). Creating this pre-prepared database of training data requires excessive work time. Furthermore, (2) the phrases (terms, etc.) contained in the training data must be identified from conversations and communications with workers during meetings. The determination of whether "items that must be confirmed" exist during meetings (conferences) such as morning assemblies held at the work site varies widely, resulting in a lack of reliability in quantifying and evaluating whether the confirmation of various items is being carried out appropriately. As a result, (3) there were few opportunities for on-site supervisors to use the evaluation results on the spot, and the utilization rate was low. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent No. 7337772 Summary of the Invention [Problem to be solved by the invention]

[0007] The present invention provides a meeting evaluation system that entrusts the creation of the above-mentioned training data, which requires time and effort, to a generation AI in order to ensure that meetings at the site are neither too much nor too little, and also entrusts the evaluation of the content of the meetings to the generation AI, making it possible to properly communicate to the site staff whether any meeting items at the site have been missed, and stores the results in the applicant company's computer and aims to use them as indicators to be useful for meetings with affiliated companies and for future meetings of the same type of construction work. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems, the invention of claim 1 of the present application provides a meeting evaluation system, which includes the steps of recording audio data of meetings held at a site such as a construction site or a building site in a communication device held by a person in charge of the site, sending the audio data recorded in the communication device to an unsafety detection PF, which is a system infrastructure on the cloud, the unsafety detection PF receiving the audio data and sending the audio data to a text conversion service on the cloud for conversion into text, the text conversion service on the cloud sending the converted text data to the unsafety detection PF, and the steps of recording the text data received by the unsafety detection PF together with meeting evaluation items for meeting evaluation and the associated evaluation items. The method includes the steps of: sending a definition prompt consisting of score evaluation criteria for each of these items to a generation AI on the cloud; the generation AI on the cloud evaluating the meeting with comments and scores from character strings present in the text data based on the definition prompt, in accordance with the meeting evaluation items and the evaluation criteria for those items; the generation AI on the cloud sending the meeting evaluation result with comments and scores to an unsafety detection PF; the unsafety detection PF4 sending the meeting evaluation result with comments and scores to a communication device held by the on-site person in charge; and displaying the meeting evaluation result with comments and scores on the communication device browser of the communication device held by the on-site person in charge. Furthermore, the invention of claim 2 of the present application is characterized in that in the meeting evaluation system described in claim 1, the definition prompt sent to the generation AI is a definition prompt that displays a score along with a comment based on the presence or absence of text (character string) that indicates an outline of the item content for each meeting evaluation item. The invention of claim 3 of the present application is characterized in that, in the meeting evaluation system described in claim 1, it has the steps of sending the meeting evaluation results to the on-site staff member's smartphone browser and sending the evaluation results to the office manager by email, and the office manager receiving the meeting evaluation results displaying the evaluation results on the manager's PC and downloading them to the office manager's PC. [Effects of the Invention]

[0009] With the above-mentioned configuration, the present invention entrusts the creation of training data, which requires time and effort, to a cloud-based generation AI, and also entrusts the evaluation of the content of the meeting to a cloud-based generation AI, making it possible to properly communicate to on-site personnel whether any meeting items have been missed, and the results can be stored on the server of the applicant company and used as an indicator to be useful for meetings with affiliated companies and for future meetings on similar construction projects. In other words, the evaluation results of morning meetings held at the work site can be obtained in a short time during the meeting time at the work site, and the evaluation results can be displayed on the browser of the smartphone of the work site supervisor, etc., so that the work site supervisor, etc. can check the meeting evaluation results and recognize the results of the meeting that day.If any items are missed, they can be immediately filled in, and as a result, the appropriateness of the meeting before work begins can be ensured. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing a first embodiment of a meeting evaluation system, which is an embodiment for implementing a meeting evaluation system according to the present invention. [Figure 2] FIG. 2 is a flow chart of the meeting evaluation system 1 according to the first embodiment. [Figure 3] FIG. 3 is a diagram showing an overall outline of a system for confirming and evaluating a meeting according to an embodiment of the disclosed invention, which is attached as FIG. 1 in Japanese Patent Publication No. 7337772. DETAILED DESCRIPTION OF THE INVENTION

[0011] An embodiment for implementing a meeting evaluation system according to the present invention will be described in detail with reference to the accompanying drawings. [Example]

[0012] FIG. 1 is a diagram showing a first embodiment of a meeting evaluation system, which is an embodiment for implementing a meeting evaluation system according to the present invention. In Figure 1, reference numeral 1 indicates an outline of the meeting evaluation system according to Example 1, 2 indicates a work site, 3 indicates a communication device such as a smartphone owned by a site supervisor or the like (hereinafter referred to as "smartphone 3"), 4 indicates an unsafety detection PF on AWS, 5 indicates a Wisper service on Azure and an OpenAI service as a generation AI, and 6 indicates the office manager's PC (computer).

[0013] Here, AWS refers to Amazon Web Services, a cloud computing service provided by Amazon, Azure refers to Microsoft Azure, a cloud computing service provided by Microsoft, Wisper refers to a speech recognition model developed by OpenAI, a transcription service, OpenAI Services is a service using generative AI, and Unsafe Detection PF4 refers to the system infrastructure related to this Example 1 built on AWS by the applicant company of the present application.

[0014] As shown in FIG. 1, the meeting evaluation system 1 according to the first embodiment places the unsafety detection PF4, which is the base system for meeting evaluation, on AWS (AMAZONN Web: Amazon Cloud Service), and also connects to OpenAI's transcription service (Wisper) via an API (Application Programming Interface) on Azure (Microsoft Cloud Service), making each cloud service executable from other cloud services, and is therefore premised on use on cloud services. However, the meeting evaluation system 1 according to the present embodiment 1 does not necessarily need to be built on a cloud service, and it is also possible to deploy separate PCs and connect these PCs together without being bound by cloud services.

[0015] However, since the meeting evaluation system 1 according to this embodiment 1 is based on the use of OpenAI, which is the generation AI, it is based on the use of OpenAI from OpenAI Inc., but this does not prevent the use of other generation AIs as the generation AI.

[0016] Under the premise of using such a cloud service, the meeting evaluation system 1 according to the first embodiment enables meeting evaluation through the following steps. FIG. 2 is a flowchart of the meeting evaluation system 1 according to the first embodiment, and the execution outline of the meeting evaluation system 1 according to the first embodiment will be explained below based on this flowchart.

[0017] First, the on-site person in charge starts the on-site meeting ((1) Start) (Step 1), and at the same time, using the browser function of smartphone 3, (2) opens Unsafety Detection PF4 on AWS in the on-site person in charge’s smartphone browser (Step 2). Next, (3) after authenticating themselves on the browser screen, they enter the organization they belong to and the number of people in the work team (Step 3). Then, (4) when the meeting starts, they press the start recording button on the browser screen (Step 4). (5) The meeting audio is recorded on the smartphone (Step 5) and the audio file is sent to Unsafety Detection PF4 (Step 6). (7) The audio file is sent from Unsafety Detection PF4 to Wisper, a text conversion service on Azure (Step 7). (8) Wisper on Azure converts the audio file to text and sends it back to Unsafety Detection PF4 (Step 8). (9) The text data from Unsafety Detection PF4 is sent to OpenAI on Asure together with the defined prompt (Step 9). OpenAI (10) evaluates the meeting based on the defined prompt (Step 10). The definition prompts sent to OpenAI and the meeting evaluation based on them will be described in more detail later.

[0018] After the meeting evaluation based on the prompts defined in OpenAI, (11) the evaluation results (evaluation points and comments for each item) are sent to the Unsafety Detection PF4 (Step 11). The Unsafety Detection PF4 (12) sends the evaluation results to the site staff member's smartphone browser (Step 12), and (13) displays the evaluation results on the site staff member's smartphone browser screen (Step 13). The site staff member then (14) re-informs the work implementers of any deficiencies in the evaluation based on the evaluation results (Step 14), and (15) concludes the site meeting (End) (Step 15).

[0019] (12) The evaluation results are sent to the site staff member's smartphone browser (Step 12), and (16) the evaluation results are sent to the office manager by email (Step 16). The office manager, upon receiving the evaluation results, (17) displays the evaluation results on the manager's PC (Step 18), and (19) stores the evaluation data in the unsafety detection PF4 (Step 18). (19) The evaluation data downloaded to the office manager's PC is saved on the company's internal file server as needed (Step 19). A prompt for a cloud-based generation AI generally refers to the characters or symbols used to prompt the user for input in a computer's command line interface (CLI), and in the case of a generation AI, it refers to a string of characters or the like that the user gives to the AI ​​to instruct it on what to generate.

[0020] From the above, the meeting evaluation system 1 according to the first embodiment is basically characterized by having the following steps. (1) A step of recording audio data of a meeting held at a construction site or other site in a communication device held by a person in charge of the site; (2) A step of sending the voice data recorded on the communication device to the unsafety detection PF, which is a system infrastructure on the cloud; (3) The unsafety detection PF receives the voice data and sends it to a text conversion service on the cloud to convert it into text. (4) A step in which the cloud-based text conversion service sends the converted text data to the unsafety detection PF; (5) A step of sending a definition prompt consisting of meeting evaluation items for meeting evaluation and score evaluation criteria for each item together with the text data received by the unsafety detection PF to a generation AI on the cloud; (6) In the generation AI on the cloud, a step of evaluating the meeting with comments and scores from character strings present in the text data based on the definition prompt in accordance with the meeting evaluation items and the evaluation criteria for those items; (7) A step in which the generation AI on the cloud sends the meeting evaluation results with comments and scores to the unsafety detection PF; (8) A step in which the unsafety detection PF4 sends the meeting evaluation result with comments and scores to the communication device held by the on-site person in charge; (9) A step of displaying the meeting evaluation results with comments and scores on the communication device browser of the communication device held by the on-site person in charge.

[0021] (Meeting evaluation by OpenAI, a generative AI) Next, the meeting evaluation in OpenAI, which is the generation AI of the meeting evaluation system 1 according to the first embodiment, will be described.

[0022] [Table 1]

[0023] Table 1 shows the definition prompts for meeting evaluation in the step (step 9) of sending a text file of the meeting audio data converted from the unsafety detection PF4 to OpenAI on Asure together with the definition prompts in the meeting evaluation system 1 according to the present embodiment 1. The above definition prompts 1 to 13 are entered according to the input screen of OpenAI, and a comment and score are displayed to indicate whether or not the corresponding character string exists in the text file. Table 1 is an example of a definition prompt that shows the "evaluation score criteria for each item" and the "full score criteria" when the criteria are met for items ranging from "evaluation items 1 (date) to 13 (safety recitation)." For example, item 1 (date) is an item that evaluates whether or not a supervisor or other person utters the date in the audio text of a meeting held on-site, and if a worker or other person utters the date, it requests a full score of "3," and is evaluated based on whether or not a character string representing the date exists in the text file. Whether or not a string represents a date is determined by OpenAI selecting strings related to dates from the wide variety of strings commonly used within its own network, and then using the presence or absence of the strings to evaluate the meeting. This is the significance of the present invention in that it eliminates the need to create training data for the invention of the prior art, Patent No. 7337772, by entrusting the creation of training data to OpenAI. The items 1 to 13 have been determined to be necessary for the safety management department of the applicant company to carry out safe work in each department's construction work.

[0024] Although detailed knowledge is not clear about the outline of the required OpenAI evaluation procedures and mechanisms, it appears that the system likely searches for information such as what a "date" is, what a "date of work" is, etc. (selecting the relevant text (character string) from all text (character strings) existing on the network), and based on the results, determines whether or not the relevant text (character string) is spoken within the audio-text file, and then issues an evaluation score and evaluation comments based on the above criteria. Also, for example, with regard to item 6 (the work to be done on the day), it seems that they search for information such as what the "specific location" is for this type of work, and "how far the work needs to be done to make it clear" for this type of work (selecting relevant text from all languages ​​that exist on the network), and based on the results, if these conditions are met, they will give a rating of "10" (the full mark), but conversely, if "only a rough process with no specifics" (not applicable linguistically), they will give a rating of "0".

[0025] (Example of meeting evaluation results using OpenAI) We will now explain an example of a meeting evaluation result obtained using OpenAI as described above.

[0026] [Table 2]

[0027] [Table 3]

[0028] Tables 2 and 3 are examples of on-site meeting evaluations by OpenAI. Table 2 shows the evaluation results for a meeting held on April 16, 2024, and Table 3 shows the evaluation results for a meeting held on May 10, 2024. In Tables 2 and 3, following the "evaluation item content" and "maximum score" for each evaluation item (1 to 13), the evaluation results show linguistically appropriate comments on whether or not the audio text contains text (character strings) that correspond to each item, and an evaluation score is displayed. One of the features of using OpenAI is the ability to obtain linguistically expressed comment evaluations and indicate which of the possible scores corresponds to each item.

[0029] As is clear from Tables 2 and 3, according to the meeting evaluation system 1 of this embodiment 1, text data is obtained from the meeting audio data recorded at the work site, and the meeting evaluation based on the definition prompt for meeting evaluation is evaluated from that text data by OpenAI, which is a generation AI. This eliminates the need to manually input and accumulate the required training data in advance, as in Patent No. 7337772 ("Meeting Confirmation and Evaluation System") of the applicant of the present application, and OpenAI evaluates the meeting content using appropriate comments and scores, so the evaluation results can be quickly reflected in on-site meetings.

[0030] In other words, according to the meeting evaluation system 1 of the first embodiment, the creation of training data, which is time-consuming and labor-intensive as in Patent No. 7337772 ("Meeting Confirmation and Evaluation System"), is entrusted to the cloud-based OpenAI generating AI, and the evaluation of the meeting content is also entrusted to the cloud-based OpenAI generating AI, enabling appropriate notification to on-site personnel regarding any omissions in the meeting items. Ultimately, the results are stored on the applicant company's server 6 and can be used as indicators for meetings with affiliated companies or for future meetings on similar construction projects. Evaluation results for morning meetings held at work sites can be obtained quickly within the meeting time, and the evaluation results can be displayed on the browser of the work site supervisor's smartphone, allowing the work site supervisor to view the meeting evaluation results and recognize the results of the day's meeting. If any items are missing, the missing items can be immediately filled in, thereby ensuring the appropriateness of the pre-work meeting.

[0031] This will eliminate the drawback of the invention proposed by the applicant and disclosed in Patent Publication No. 7337772, which was unable to instantly make up for shortages at the work site on the day. In the meeting evaluation system 1 according to this embodiment 1, the unsafe detection PF4, which is the system infrastructure according to this embodiment 1, has been described as being on the AWS cloud, and the text conversion Wisper and the generation AI OpenAI are each on a separate Azure cloud, but it goes without saying that these may be constructed on the same cloud, or that individual computers and servers may be prepared and connected in-house without using the cloud. [Industrial Applicability]

[0032] The present invention is used in a meeting evaluation system. [Explanation of symbols]

[0033] 1. Overview of a Meeting Evaluation System According to a First Embodiment of the Present Invention 2. Work site 3. Communication devices such as smartphones owned by site supervisors, etc. 4. Unsafety detection PF on AWS, which is the base system according to the first embodiment 5 Wisper service and OpenAI service on Azure 6. Office manager's PC (computer) 101 Meeting confirmation and evaluation system according to Example 1 of the disclosed invention of Patent No. 7337772 102 Work Site 103a Conference audio file 103b Instruction signal 103c Work details data 104 Work Management System 105 Computer 106 monitors 107 Output results

Claims

1. A step of recording audio data of a meeting held at a site such as a construction site or a work site in a communication device held by a person in charge of the site; A step of sending the voice data recorded in the communication device to an unsafety detection PF, which is a system infrastructure on the cloud; The unsafety detection PF receives the voice data and sends the voice data to a text conversion service on the cloud to convert it into text; A step in which the text conversion service on the cloud sends the converted text data to the unsafety detection PF; A step of sending a definition prompt consisting of meeting evaluation items for meeting evaluation and score evaluation criteria for each item together with the text data received by the unsafety detection PF to a generation AI on the cloud; A step in which a generating AI on the cloud evaluates a meeting with comments and a score from character strings present in the text data based on a definition prompt and in accordance with the meeting evaluation items and the evaluation criteria for those items; A step in which the generation AI on the cloud sends the meeting evaluation results with comments and scores to the unsafety detection PF; A step in which the unsafety detection PF4 sends a meeting evaluation result with comments and scores to a communication device held by the on-site person in charge; displaying the meeting evaluation results with comments and scores on a communication device browser of the communication device held by the on-site person; A meeting evaluation system comprising:

2. The meeting evaluation system described in claim 1, characterized in that the definition prompt sent to the generation AI is a definition prompt that displays a score along with a comment based on the presence or absence of text (character string) that outlines the content of each meeting evaluation item.

3. a step of transmitting the evaluation result according to claim 1 to a smartphone browser of the on-site staff member and sending the evaluation result to an office manager by email; a step in which the office manager who has received the evaluation results displays the evaluation results on the manager's PC and downloads them to the office manager's PC; 2. The meeting evaluation system of claim 1, further comprising:

Images